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ABSTRACT 

The purpose of this mcnograph is to stimulate 
reflective inguiry by educators into their work as policy makers, 
program designers, and evaluators of inservice education programs. 
Its context is a series of case studies of evaluations performed in 
Denmark, Sweden, the Onited Kingdom, and the Onited States. Four 
essays are presented. The first conveyed, on practical grounds, the 
notion that reflecting upon previous experience in evaluating 
inservice education is valuable. Future performance will depend upon 
the extent of understanding of previous work. In the second essay, 
the point is made that, if experience in evaluation is to become 
valuable to future perforwance, sound evaluation principles must be 
established. The third essay related in some detail those issues that 
the writers of the case studies felt were major problems in 
performing their evaluation studies. The final essay suggested that 
reflectivity is not only a preliminary activity leading to 
evaluation, but may be essential during the entire svaluation 
process. (JDl 
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I. Why Reflect Upon the Evaluation of 
Inservice Education, and Training 



A. Why Reflect 



The purpose here is to ask questions of ourselves. We 
are evaluators from different countries, from differing tradi- 
tions of education, public policy and professional development. 
Our status has been fluid, our professional milieu has been 
complex. In addition to performing our own evaluations, we 
have described and reviewed ot*her Deople ! s owrk in evaluating 
inservice education and training of teachers (INSET). We 
are also teacher educators who have designed educational 
activities for teachers and other educators. We have at times 
worked closely with - or have "been ourselves - middle level 
managers within a bureaucratic agency responsible for monitor- 
ing or carrying out inservice education programs for public 
policies. Some of us have worked with elected officials who 
make policy on inservice education and training in our 
respective countries. In short, our professional roles and 
corresponding perspectives are in a state of flux, making 
answers to questions about our work in INSET evaluation transient. 



This leads us to why we muse ask questions of ourselves. The 
complexity and fluidity of our roles in inservice education and train- 
ing make it necessary for us to take periodic stock of what we know. 
What we have learned in our respective professional lives is 

not always apparent to us and seldom made known to others. The heuris- 
tics of our experience leave us little choice but to reflect upon what 
we have done, why we have done that and how experience has changed our 
perspectives on inservice education and training of educators. There 
are few educators or general publics in our respective countries who 
are much interested in inservice education and training of educational 
personnel. If critical analysis of our work is to be done, if stimula- 
tion for better performance in evaluating inservice education and train- 
ing is to be provided, it will have to be provided by us who do the work. 
The underlying reason why we must reflect upon our own performance, how- 
ever, is that we are uncompromisingly responsible for our own acts in 
evaluating inservice education and training programs. The 
nature of our professional lives suggests that we have an unusual degree 
of flexibility in reforming our work. In the contexts of our own respec- 
tive countries, there is no doubt we can do better next time if we so 
desire. 



If we consider educational evaluation (rather than inservice educa- 
tion and training), arguing for reflectivity into our performance is 
just as compelling but for different reasons. Educational evaluation 
is receiving renewed professional interest in our respective countries. 

In the United States the past few, years have been a dynamic period 
in evaluation as indicated by major changes in the perspectives of 
recognized leaders in educational research and evaluation. In Denmark, 
while inservice education and training has been, performed by a national 
agency for over 100 years, program evaluation of that enterprise is just 
gaining attention as a professional craft. In Sweden, where major edu- 
cational and political reforms have occurred in the past forty years, 
evaluation is being recognized as a necessary instrument for formulating 
and implementing educational policies. In Great Britain, the evaluation 
work of persons such as Parlett, Hamilton, MacDonald, Elliot, Adelman and 
Walker has increased not only a professional focus on the art of evalua- 
tion but on the general nature of educational research — its philosophic- 
al, political and educational foundations. In France, evaluation is 
considered as educational research where notions of science and its 
relationship to the reform of practice become major grounds for debate. 
In Australia, evaluation is booming to the point where it is nearly 
becoming the new science of education. Investigations into the induction 
period (the first few years of teaching), for example, have gained much 
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attention within Australian educational policy networks. 

In all of these countries, there is no single view of educational 
evaluation that is considered to be relevant to all evaluation condi- 
tions. No country, for example, is considered to be a significant 
modeler of evaluation practice. Evaluators in each country are looking 
instead to their own cultural, political and educational contexts for 
practices that can be adapted to specific evaluation challenges. 
Evaluators are asking for international sharing oaly to the extent to 
which they can build their own eclectic styles relevant to their own 
national contexts. The focus on context goes within national boundaries 
and into specific programmatic settings and the political milieu in 
which particular educational policies are made and carried out. 

This Very brorf sketch of the current state of evaluation implies 
a significant need for self-reflective behavior by us who practice or 
support evaluation in our own countries. We do not expect to adopt any 
particular evaluation model, perhaps not even the model we used last 
year. Instead, we intend to adapt features of different models to par- 
ticular professional challenges as we investigate our activities and form 
our educational policies towards inservice education and training. 
Adapting models from our past experiences and from the distillations of 
others' experiences requires a level of understanding that is reached only 
through critical, analytical inquiry into our own work. 

B. The Purpose and Context of This Monograph 

The purpose of this monograph is ' 5 stimulate reflective inquiry 
into our work as policy makers, as program designers and as evaluators 
of inservice education and training programs. By reflective inquiry we 
mean analyzing our work, our own personal experience and performance, 
with the aim of making our future work better. The intent of this mono- 
graph is to redirect our critical attention from the work of others to 
the work we do as policy makers, program designers and evaluators in 
evaluating inservice education and training. Reflective inquiry into 
the evaluation of INSET programs includes analyzing our intentions for 
inservice education and training, reviewing the principles which guide 
us in designing evaluations of INSET programs and clarifying the issues 
that have arisen from our experience in implementing these evaluation 
designs. Reflective inquiry also includes making recommendations to 
ourselves for redirecting our future work in evaluating inservice edu- 
cation and training programs. 

The context of this monograph is a series of case' studies 
of INSET evaluations performed in four countries; Denmark, 
Sweden, United Kingdom and United States and selected cases 
of INSET evaluations performed in France and Australia. Along* 
\yith the case.. studies were a series of international conferences 
on INSET sponsored by the "Centre' for "EducatI6rial"Res*earc7i - *"a^ 
Innovation of the Organisation for Economic Co-operation and 
Development (Paris), in co-operation with tlje National Institute 
for Education (U.S.), the U.K. Department of Education and 



Science, the~"' Swedish Ministry of Education, the Swedish 
National Board of Education and the Palm Beach County 
School System (U.S.)* These conferences were held in 
Philadelphia (June 1976), Stockholm (October 1976), West 
Palm Beach (November 1977) and Bournemouth (March 1978). 
In addition to the international conferences and the national 
case studies on the evaluation of inservice education and 
training,, a preliminary draft of this report was written and 
a conference held with the principal writers Of the case 
studies in November 1978, in Paris. This monograph is a 
rev/rite of that report based upon the issues raised by the 
case study writers. Thus, the context for this analysis is 
international, longitudinal and grounded in the field experience 
of those who evaluate, design programs, and help form national 
policies on inservice education and training in their respec- 
tive countries. The intent of this monograph is to tap \this 
resource of field experience from different' national 
perspectives. 

C. The Readership and Format For This Monograph 



This monograph is written for those who have performed or supported 
evaluations in the past and expect to be linked to evaluations of inser- 
vice education and training in the future. This may include a diverse 
but quite limited group of people. '"Diverse" because those who see 
inservice education and training as a way to improve school performance 
can include elected officials, middle-level managers of public agencies, 
teachers, teacher educators, school administrators, educational research- 
ers and educational program evaluators. "Limited" because there may be 
relatively few in these roles who include a vision of a better workplace 
for the adults who educate our children along with their visions of 
improved schooling. Thus, the monograph is written as a challenge to 
those policy makers, program designers and evaluators who are convinced 
that opportunities for educators 1 professional development are necessary 
if we are to nourish or improve the education of our young. 

Policy makers include governmental agency heads and middle-level 
managers who have the responsibility for implementing a legislated 
policy in education. It is they who may determine that inservice ed- 
ucation and training is a reasonable means to achieve a particular edu- 
cational policy. Policy makers may include elected officials who see 
inservice education and training as a viable means for achieving 
educational or social aims. They may also include public agency admin- 
istrators who are responsible for articulating official views of what 
professional development means and what certification is. Program 
designers include teacher educators based at universities, colleges 
or government agencies who have had responsibilities for educating stu- 
dents before they enter teaching or have taught teachers in graduate 
courses or other school settings. Program designers may also include 
those school personnel who articulate school authority policy into 
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specific teacher training designs. Evaluators include those educators 
with professional reputations in educational research but with outside 
interests in program evaluation. They may be researchers (often from a 
teacher education institution) who consider the evaluation of inservice 
education and training as an interesting arena to apply their skills. 
A few may be full-time evaluators emplqyed by a private firm relying 
on its capacity for gaining work. Some evaluators may be employed full 
time by a government agency, school system or university. 

Although there are others who may become responsible, the evaluation 
of an inservice education and training program is usually at the discre- 
tion of the policy maker, program designer or evaluator. The purpose 
in addressing an audience of policy makers, program designers and evalu- 
ators is that the success in evaluating future inservice education and 
training programs depends upon a mutually shared ability to ask the tough 
questions. The following format has been chosen to engage this varied 
audience. 

There are four parts to the monograph , 'each is a brief essay 
written for consideration by policy makers, program designers and eval- 
uators. The four essays are: 

• Why reflect upon evaluations of inservice education 
and training; 

© Principles do not serve us well: a critique based 

upon international case studies; 
O Issues to confront when reflecting upon the evaluation 

of inservice education and training; and 
•Reflecting in context: confronting wisdom and 

ignorance in ourselves and in our colleagues. 

The f-Sr-st essay ("Why reflect") is meant to convey on 
practical grounds, the notion that previous experience 
in evaluating inservice education and training is valuable. The point 
is made that we need Co recognize that our future performance will de- 
pend upon the extent to which we understand our previous work. This is 
not, it will be pointed out in the following section, an ability that 
many of us have shown. The intent of this essay is to generate respect 
for our past experience in evaluating inservice education and training 
while not diminishing the need to do better in the future. 

The second essay ("Principles do not serve us well") is a critique 
of six principles formed at a recent international conference on 
school-focused inservice education and training. The 
basis for this critique is the national surveys of case studies on the 
evaluation of inservice education and training (mentioned previously}*. 
This essay continues- the respect for previous 
experience in evaluating inservice education and training programs, 
referring to the case studies as documentary evidence for some of that 
experience. The essay argues that our purposes for inservice education 
and training and our experience in carrying out these purposes is far 



richer than the"~principles we usually generate in public meetings, . 
The intent of the essay is to make it clear that if our experience is 
to become valuable to future performance, we must do better than ex- 
tracting, general principles that seem to aggregate across national 
and international contexts. 

The third essay ("Issues to confront 11 ) relates in some detail 
those issues that case study writers felt were^ major problems in 
performing their evaluation studies. Some of these issues were raised 
in the national surveys, but most were raised in the discussions and 
deliberations of the case study writers' seminar in Paris (November, 
1978). The essay shows how reflectivity into past performance can 
bring up those crucial topics that are essential to address but diffi- 
cult to resolve and perhaps even more difficult to recognize. The essa 
suggests one way to recognize these issues in our work is to consider 
three realities in evaluating INSET; the bureaucratic context, the 
choice of methodology and the meanings of participation. The intent of 
this essay is to make it more clear what is at stake as we become in- 
volved in evaluating inservice education and training programs. 

The final essay ("Reflecting in context 11 ) goes further with the 
theme of reflectivity, suggesting it is not only a preliminary activity 
leading to evaluation of inservice education and training programs but 
may be essential during the entire evaluation process. With this in 
mind, it is necessary for each of us involved in an evaluation of inser 
vice education and training — policy makers, program designers and evalu 
ators — to analyze our own understanding on topics, some of which we may 
not have thought about for years. The essay includes recommendations 
to policy makers, program designers and evaluators on what their respon 
sibilities vould be if they were to reflect upon Cheir own experience 

and understanding and confront others' intentions accordingly. 

The intent " of this~ essay*" is f o suggest how we can' make 
our e::perience and understanding v/ork for us as we 
evaluate particular inservice education and training 
programs . 
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D. Recognizing Evaluation and Inservice Education 
and Training in the Context of CERI/OECD 



Earlier it was argued that reflection is necessary because of our own 
Varied professional development, the absence of much interest and 
stimulation by outside critics of inservice education and training, 
and the contrasting state of interest and stimulation in program eval- 
uation at this time in our respective countries. Such arguments may 
be well and good for evaluators and program designers but do not 
say much to agency heads and middle level managers of education. 
They nay also be less useful to 'OECD/CERI which . 

has developed this program in order to increase" 

interest in~ ins'ervice education 

and training because of the "role ~" INSET plays' in improving 

school practice in member countries. Reflecting, that is reviewing one's 
performance in terms of a variety of perspectives (e.g. one's intentions, 
official purposes, what actually happened), can seem to be a relatively 
passive undertaking compared to the design and implementation of new * 
programs. Although we know that current and future performance is not 
independent of the lessons we have learned from past performance, the 
challenge of the manager is to get something done. Lessons will be 
learned by those who practice. Evaluators and program designers have 
responsibility for reflection on evaluation and inservice education and 
training. The manager's responsibility is to make sure that performance 
meets the objectives, in this case, to see that evaluations of inservice 
education and training help improve schools of OECD member countries. 

That view of reflectivity is wrong and events sponsored by CERI/OECD 
provide us with good examples of why it is wrong. One example is found 
in the traditional focus of the Centre for Educational Research and In- 
novation (CERI) on school innovation. The focus of CERI on curriculum 
reform parallelled comparatively great efforts in most member countries ' 
to improve the quality and equality of their schools by concentrating 
resources into the design of curricular programs. In the 1960 f S 
and early 70 ! s, participants from all member countries and CERI knew 
innovation was curriculum reform and that one necessary feature of 
curriculum reform was to get the curriculum into the schools. The more 
recent emphasis of CERI on inservice education and training is a direct 
outgrowth of these past performances in curriculum reform and the lessons 
that have been learned from that performance. Educational managers 
understand, for example, that the major problems in the past era were 
not in curriculum design but in getting those designs into school prac- 
tice. Given the lessons learned from that period it is natural and wise 
for educational policy makers (in CERI and in OECD member countries) to 
turn their attention to the potential of inservice education and training 
as a crucial factor for implementing educational policy. As much as 
CERI policy makers may realize the historical antecedent and relationships 
of inservice teacher education and training to the curriculum reform 
efforts performed under the rubric of innovation, it is clear that this 



understanding ±s not shared by evaluators or designers of inservice 
education and training. Nothing in the reports of past conferences 
on inservice education and training sponsored by CERI, 
nor in the recent CERI publications on inservice education and 
training link INSET to experiences in curriculum reform. We evalua- 
tors and program designers look at INSET as a new 1 area to apply our 
craft rather than as a factor linked to past failures in improving 
our schools. 

This tendency not to reflect upon past performance is especially 
pronounced in certain reports and publications dealing with the evalu- 
ation of inservice education and training. A repeatedly expressed 
opinion that the evaluation of INSET is a new activity that may (or 
may not) borrow its methods from curriculum evaluation is an example 
of the misunderstanding we evaluators have about our past performance. 
Many past evaluations of curricular programs addressed problems in 
implementing these programs in the schools, focusing upon teachers, 
teacher trainings , .teacher education and school context. Many evaluators 
who are cited as pioneers in curriculum evaluation were forming their 
approaches to understand problems in transferring a designed curriculum 
into the schools. Alternatives created by Parlett and Hamilton (1972), 
MacDonald and Walker (1974), Elliott (1977), Berliner et. al (1978), 
House (1974), Stake (1974) and Scriven (1974), for example, are erro- 
neously referred to as reforms in the evaluation of curriculum. Instead 
they created approaches to better understand how a curricular program 
is integrated into the schools. They focused on one major purpose of 
INSET (to improve school practice), as well- as on the educational and 
training processes used with school personnel. This lack of attention 
by evaluators to the historical context of inservice education and 
training and of the alternative forms of evaluation produced in that 
context suggests that professional reflection upon past performance does 
not occur naturally. 

.There are two. important conclusions that policy makers can make 

from these examples. The first is that if professional reflection by 
evaluators and/or program designers is warranted or seems desirable, 
it will have to be directly supported and encouraged by the policy makers 
Even when it seems obvious to the policy maker that past performance is 
related to current intentions, that may not be so obvious to either the 
evaluators or to the program designers. The second conclusion is perhaps 
even more crucial, and that is that evaluators and program designers can 
not be expected to reflect alone and draw relevant lessons learned. 
There is information held only by the policy makers that is crucial to 
a fuller understanding of why inservice education and training is deemed 
important and the role that evaluation may play. Thus, for policy makers 
the most serious conclusion is that reflectivity is their responsibility 
as well as the responsibility of the evaluators and program designers. 
Furthermore, this reflection 'by the policy makers should be done with 
the evaluators and program designers since it is their fundamental un- 
derstanding of context and purpose that must be improved. 
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In the CERI context, this would imply that there 
is much of value in CERI managers and agency heads 
reviewing the evolution of their concerns about inservice 
education and training (and its evaluation). It would also 
imply that this reflection should at some time include 
those others who have been asked to reflect and draw 
implications from their experience in evaluation and/or in 
designing inservice education and training. 
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II. JPrincip2.es Do Not Serve Us Well: A Critique 
Based Upon International Case Studies 



A. Purposes and Settings of Inservice 
Education and Training 



First, let us remind ourselves that inservice education and train- 
ing has received attention for three quite different reasons. One 
reason is its potential for stimulating professional development; a 
second is that inservice education and training can improve school 
practice, and a third is that inservice education and training may be 
a viable strategy for implementing social policy. We have all heard 
one or all of these reasons used to establish why particular inservice 
education and training programs should be supported* We have used 
them ourselves. The three reasons have been repeated often enough where 
we must take ourselves seriously, that is as if the purposes we give 
were contextual features of an inservice education and training program* 

When we support an inservice education and training program on 
the grounds that it may stimulate professional growth, for example, 
we have placed that program into a context with particular characteris- 
tics. One obvious characteristic is that the program is focused on 
individuals 1 professional advancement* The goals and objectives of the 
program are intrinsic to the goals and objectives of participating edu- 
cators. An underlying assumption is that the complexity of professional 
demands on educators and the changing social context of schooling make 
growth a necessity for professional survival. It may or may not assume tha 
knowledge about education is expanding; it does assume, however, that 
educators need to continually improve their range of professional under- 
standing and personal performance as they mature, develop and practice 
their craft. Critical to the context created by reasoning that inser- 
vice education and training will stimulate professional development are 
the theories we hold about development. These may, for example, be 
held as psychological theories of stages (e*g* Sprinthall), sociological 
theories of socialization (e.g. Lortie), or some eclectic theory of how 
teachers develop (e.g. Fcx). Each theory of professional development 
can imply very different inservice education and training strategies, 
thus making the theory of professional development a significant feature 
of these programs. 

The practice of inservice education and training to stimulate 
professional development has a tradition in our respective countries. 
In some of our countries this tradition may jump back a number of decades. 
In the United States, for example, inservice education and training for 
professional development was practiced by some followers of John Dewey 
in the igSO's and 40 f s (see Eeiman, 1979). This practice has shown 
that when the goal is to stimulate professional growth, the means may 
be the end, the performance may be the goal. A significant charac- 
teristic of the context created by referring to the professional 




development of those educators who are involved in an inservice educa- 
tion and training program are the positive connotations of "prof ession- 
al development"* It implies we can and do change (for the better) 
over time, it suggests that there are steps we can take to improve our 
own professional situation; it respects the professional autonomy of 
those who are developing. Perhaps the attractive connotations are. why 
we currently hear professional development being used in our respective 
countries as a primary reason for inservice education and training 
programs. It makes "professional development" attractive rhetoric. 
When inservice education and training is practiced, however, it is 
other characteristics such as the theoretical basis and the experience 
from past efforts that are significant to the design and evaluation 
of a program intended to stimulate professional growth. 

A different context is created when we reason that an inservice 
education and training program is intended to improve particular aspects 
of school practice. Here the inservice education and training is ini- 
tiated and motivated by a noted inadequacy and failure in* the school 
system. Inservice education and training is for an end that goes beyond 
the individual educator's well-being; it is intrinsic to the school 
system, extrinsic to the participating educator. Inservice education and 
training designed to improve a particular feature of school practice (e.g. 
a curricular program) focuses on isolated features of- participating 
educators 1 professional practice; the complexity of anticipated roles is 
largely irrelevant. It usually assumes knowledge is expanding and, in 
fact, is often designed to transfer knowledge gained from research to 
those who are practicing in the schools, It often assumes that knowledge 
about educational practice is generated only in research settings. It 
is oriented to immediate performance (and those understandings that may 
lead to immediate performance) since the intent is to resolve current 
problems. The theoretical basis for the context created by reasoning 
that inservice education and training can accomplish improvement is 
in organizational development and institutional change. Theories of 
how schools change influence the design and evaluation of these inservice 
education and training programs (e.g. McLaughlin and Herman, McLaughlin 
and Marsh, Corwin, Meyer, Fullan, Miles in the U.S.). 

There is a rich and comparatively recent tradition of inservice 
education and training for school improvement in most OECD member coun- 
tries. This tradition is found is the surge of curriculum innovations 
performed in the 1950 f s, 60' s, and 70 ! s where inservice education and 
training was a major strategy for implementing designed innovations into 
the schools. There is little reference to this tradition in recent 
discussions on inservice education and training, perhaps because of the 
unattractive connotations. The unattractive connotations include im- 
plications that greater knowledge is held by those outside the system, 
that external pressure is necessary and that thfc participating educators 
in the inservice education and training are "deficient" in some signifi- 
cant aspect. There is irony brought about by our recent past in curric- 
ulum reform. The irony is that there is considerable practical exper- 
ience in inservice education and training from tha curriculum reform 
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eras of OECD member countries that is not being acknowledged by those 
who reason that inservice education and training can improve school 
practice. 

A third context is created when we reason that a social policy is 
to be implemented through an inservice education and training program. 
This reason for inservice education and training has seldom been stated 
but often has been implied in recent OECD conference discussions and 
publications. Implied or stated, it does, however, create a third 
context where social policies are the focus of attention rather than 
the school systems or individual educators. Inservice education and 
training designed to achieve a social policy (e.g. in the U.S., deseg- 
regation) is intrinsic to legal goals of the society and may be extrin- 
sic to institutional goals of the schools or to personal goals of the 
participating educators. Professional demands and the complexity of 
professional roles may not even be relevant, except as barriers to 
social reform. The focus is on the role of education in inducing social 
reform, not only in responding to social changes. Professional 
knowledge of educators is of tan challenged in this context, particularly 
the notion that knowledge about education is expanding. It assumes that 
the understanding and professional performance of educators is related 
to the maintenance of social ills as well as to their eventual correc- 
tion (e.g.' there is personal and institutional culpability for sexual 
and social inequalities in schooling). The theoretical basis for the 
context created by reasoning that inservice education and training can 
implement social policy is found in theories of social reform. Theories 
of how social patterns in our respective countries change and how social 
reforms do or do not occur can shape the design and focus of the inser- 
vice education and training programs. These theories, for example, may 
be economic (e.g. Young), sociological (Bernstein), intellectual (Toulmin 
Habermas) or some combinations of these theories (e.g. Apple). 

Inservice education and training is considered as a means where 
achieving social policy is the end. To be more precise, inservice 
education and training is one means among many where law, financial 
incentives, architecture or community planning, for example, may be 
other means to achieve a particular social policy. Historical analysis 
suggests, that a tradition of inservice education and training for im- 
plementing social policy has happened at particular moments in our 
respective countries. One example is the industrial revolution and 
the formation of Lancastrian schools (1 teacher, 1 class, 1 subject). 
The connotations in reasoning that inservice education and training 
be used to implement social policy are both attractive and risky. The 
attractive connotations include the implications that education is a 
social responsibility, that in both an institutional and personal 
sense, educating the young is a noble and socially relevant activity. 
At the same time, we consider the view risky because of our ignorance 
of social theory and our corresponding inability to deal with funda- 
mental political dissent — whether we be policy maker, program designer 
or evaluator of inservice education and training programs. 



To summarize this discussion on context, we give 3 reasons for 
our support of inservice education and training of educators: 

1) to stimulate professional development, 

2) to improve school practice and 

3) to implement social policy. 

Each reason creatps a different context with its own set of characteris- 
tics. Figure 1 > summarizes the characteristics. In addition 
to acknowledging that we do create different contexts by our reasoning 
for inservice education and training, we must also acknowledge that 
the contexts we create are often more complex than the columns in Figure 
1 suggest. This happens when we use more than one reason to support 
a particular inservice education and training program. 

Although it does happen that we support inservice education and 
training for primarily one reason (for stimulating professional devel- 
opment, for example), it is just as often that we use multiple reasons 
for the same inservice education and training program. There are in- 
service education and training programs that we support on the grounds 
that they will stimulate professional development while improving par- 
ticular features of school practice. These are programs that we support 
on the grounds that they will implement a social policy while they 
improve a particular feature of school practice. In combining two or 
more reasons for inservice education and training, we create some of 
the more vexing (and interesting) problems for designing and evaluating 
these programs. As Figure 1 suggests, if we use two (or three) reasons 
for supporting inservice education and training, we place the program 
into two (or three) different contexts. Contradictions and conflicts 
can then occur between many characteristics of the multiple contexts. 
If we reason that an inservice education and training program will be 
performed to stimulate professional development and to improve school 
practice, for example, we are implying that the program will be in- 
trinsic to the participating educator and to the system, it will 
recognize complex unanticipated professional demands and will isolate 
significant features of practice in order to improve immediate system 
failures. 

The point is not that these conflicts are necessarily irreconcili- 
able, but that we need to give them attention when designing and evalu- 
ating such programs. Experience has shown it is possible. Three of 
the most interesting international case studies evaluated inservice 
education and training programs that were intended to stimulate profes- 
sional development and to improve school practice: Elliott (U.K.), 
Larsson (Sweden), and 01 Sen (Denmark) . Some of the most insightful 
literature in evaluating inservice education and training programs in- 
tended to improve school practice and to implement social policy are 
in the works of the Centre for Applied Research in Education in Norwich, 
England (see MacDonald; MacDonald and Walker; MacDonald and Norris; 
Jenkins et. al). Some of the richest experience and resulting insights 
we hold are in our resolutions of the conflicting chajacteristics im- 
plied in Figure 1. what we need is a better way to discuss, compare and 
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luservice education and training for: 



ST IHULAT I NG I M PUOV I NO 1 MPLEMENT I NO 

PROFESSIONAL DEVELOPMENT SCHOOL PRACTICE SOCIAL POLICY 



•intrinsic to participating 
educators 



•recognizes complex 
professional demands 



•recognizes social pressures 
for educational practice 
may change 

• may or may not assume pro- 
fessional knowledge is expanding 

•assumes that range of under- 
standing and professonal 
performance must develop to deal 
with unanticipated futures 



intrinsic to educational 
system; extrinsic to 
participating educators 

isolates signi f leant 
features for particular 
purposes 

an institutional response , 
to a particular pressure 

usually assumes professional 
knowledge Is expanding 

assumes that understanding 
and professional performance 
must be Improved to deal with 
Immediate Austltut lonnl 
failures 



intrinsic to societal 
goals, ideals; extrinsic 
to educational system 

professional demands, 
complexity not relevant 
(except as barriers to 
social reform) 

recognizes a symbiotic 
role for education in 
changing social pressure 

expanding of professional 
knowledge Is often questioned 

assumes that understanding 
and professional performance 
are related to social ills 
and can be transformed for 
social reform 



•theoretical basis In develop- 
mental theories (e.g. psychologi- 
cal stages, process of sociali- 
zation) 

•rich tradition in practices 
of 1930*8 and 1940's 



theoretical basis In organi- 
zational development and in- 
stitutional change theories 

rich t rarilt Ion In practices 
of 1950's, 60»s and 70 f s 



theoretical basis in theories 
of social reform (e.g. 
economic , political 9 
Intellectual, ideological) 

rich tradition in practices 
of certain ages of social 
reform (0,5. industrial 
revolution) 



•luservice education and 
training can be both 
means and end 



luservice education and 
training Is means to an 
Inst 1 tut lonal end 



Inservice education and 
training (indeed all of 
education) is meant) to 
social end 



•has attractive connotations 
about personal and profes- 
sional autonomy of partici- 
pating educators to Improve 
self — thus, confidently 
stated 



has unattractive connotations 
about external pressure and a 
deficiency model of the partici- 
pating educator — thus, stated 
with caution * ^ 

9 -> ^ lO 



has attractive connotations 
of social responsibility but 
accompanied by Ignorance of social 
theory and inability to deal with 
political dissent — thus, stated 
hardly at all 



Figure 1. Characteristics of Three Contexts For luservice Education and Training 

(Formed By Our Intentions) 



analyze those practices, insights and experiences we have gained from 
inservice education and training programs that have been supported for 
multiple reasons. I have two suggestions; 

1) Use a taxonomy that allows for combinations of 
the reasons to be discussed, and 

2) Use language that allows us to distinguish 
between purposes and aims of inservice ed- 
ucation and training. 

The following 3-dimensional taxonomy is suggested. 

Take the three reasons given for the support of inservice education 
and training: 

1) stimulating professional development, 

2) improving school practice, and 

3) implementing social policy. 

Consider each as a singular dimension, a continuum with intervals from 
low to high. Place the dimensions as orthogonal axes to define a space 
where an inservice education and training program is located by its 
place on the respective dimensions. 

If we take stimulating professional development and improving school 
practice as two axes, for example, the two contirma define a two dimen- 
sional space. All inservice education and trainirgprograms can be 
placed in this space. In Figure 2 INSET program A is 

primarily supported for improving school practice and minimally supported 
for stimulating professional development, INSET program B is supported 
for stimulating professional development and for improving school prac- 
tice, and INSET program C is exclusively supported for stimulating pro- 
fessional development. (The three are relatively "high 11 in their primary 
concerns which would not be necessary for all programs). 

The example in Figure 2 differs significantly from a 1978 OFm 
monograph written by Professor Ray Bolam. In that monograph, Bolam 
suggested that stimulating professional development and improving school 
practice be considered as opposite end points on the same continuum as 
is shown in Figure 3. (Bolam called the first "personal education 11 and 
the second "vocational/career education/training 11 .) 

Figure 2 and Figure 3 are similar in one respect: they both con- 
sider stimulating professional development andimproving school practice 
as continua (that is, with varying gradations of emphasis). The differ- 
ence is that Figure 2 suggests that a program can be supported for both 
reasons, Figure 3 does not. Taking INSET program C from Figure 2, for 
example, and placing it on the Bolam grid (Figure 3) is simple. Perhaps 
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Figure 2: Example of 2-dim- 
ensional space for locating 
inservice education and train- 
ing programs 
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Figure 3. One Dimension ofBolam GridfOECD, 1978) 
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placing INSET program A on the Bolam grid is not much of a problem, 
either. Where, however, would INSET program B be placed on the Bolam 
grid? Does the Bolam grid even allow for the existence of INSET 
program B, a program intended to improve school practice and to stim- 
ulate professional growth? Figure 4 includes the third reason for 
supporting inservice education and training, to implement social policy, 
as a third dimension. Figure 4 is a 3-dimensional space where inservice 
education and training programs can be located by the reasons expressed 
for their support. 

In addition to the 3-dimensional space of Figure 4, I suggest that 
it is necessary at times to distinguish the purposes from the aims of 
our inservice education and training programs. Purposes are our im- 
mediate intentions and aims are our eventual goals for inservice educa- 
tion and training. If wk permit a distinction between purposes and 
aims, then we can refer, for example, to a program whose purpose was 
to stimulate professional development and whose aim was to improve 
school practice. For such a program it may be essential to ask in what 
ways the purpose (stimulating professional development) was related to 
its aim (improving school practice). A distinction between purpose and 
aim would allow us to describe and analyze (from the performance of the 
program) the extent to which its aim was reached and the extent to which 
the purpose, in this instance, may be judged to have been related to the 
aim. There may be many inservice education and training programs where 
distinctions between purposes and aims are essential in evaluating the 
performance and success of the program. 

A significant part of the context of our inservice 
education and training programs is created by the reasons 
we use for supporting our programs. These contexts may be 
different, depending upon our reasons (see Figure 1); they 
may be complex when we have multiple reasons or when there 
are distinctions between the purposes and the aims of 
our programs. The complexities of our programs may be more 
capable of analysis, comparison and understanding if a) we 
consider seriously the characteristics implied by the three 
reasons we use to support inservice education and training, 
b) we use a 3-dimensional taxonomy that allows for combina- 
tions of these reasons to be used for the same program 
(see Figure 4), and c) we use a language that can distinguish 
our reasons for inservice education and training into purposes 
and aims when it is appropriate. 
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Another significant part of the context of our inservice education 
and training programs is the setting in which a program takes place. 
The international case studies suggest that there are three different 
settings in which inservice education and training programs h^ve been 
performed: 

!) in single schools , for the entire staff, some- 
times referred to as "school-focused 11 INSET, 

2) in multiple schools , for entire or selected 
staff from an educational system larger than 
a single school (e„g. a "feeder system 11 or a 
"school district" in the U.S.), and 

3) in ad-hoc groups , for persons in a group which has 
been formed on a temporary basis (e.g. a university 
course or a series of Teacher Center workshops). 

These three settings (single school, multiple school, ad-hoc 
groups) can be considered as strategies for reaching the purposes (and 
aims) of our inservice education and training programs. 

Thus, there are two significant parts to the context of our inser- 
vice education and training programs: 1) that created by the three 
reasons we use to support inservice education and training and 2) the 
settings in which these programs are performed. To articulate or anal- 
yze our experience, to judge our effectiveness, to 
assess what we have learned from our experience in inservice 

education and training will take a precise identification of what our 
intentions are and what the setting is. If we discuss only the setting, 
for example, important features of our own experience are left uncriti- 
cally analyzed and resulting professional dialogue becomes disjointed 
and unilluminative. 
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B. A Critical Analysis of Six Principles Formed ac Bournemouth 



"In the development of school-focused INSET, evaluation was agreed to 
be: essential; collaborative, with teachers involved; important for 
diagnosis, process study and outcome measurement; part of the teachers 1 
professional responsibility and required at many levels within schools 
as organizations. " 

(Bournemouth, England, March, 1978 as 
reported by Mr. Keith Baker, p. 13) 

Keith Baker's report on the Bournemouth Conference offers a succinct 
synopsis of five characteristics that conference participants agreed 
were basic to the evaluation of school-focused inservice education and 
training. In addition to the five characteristics described above, 
there was a sixth characteristic that all agreed was necessary in evalu- 
ating school-focused INSET: that the investigations be "illuminative" 
in the sense described by Parlett and Hamilton. 

There were 42 participants at Bournmouth from a number of OECD 
member countries. They were to address their remarks to school-focused 
inservice education and training. They were to focus on two issues, 
one being strategies for evaluating school-focused inservice education 
and training, the other being the training of INSET trainers. One pur- 
pose of the Bournemouth meeting was to identify where international 
collaborative efforts in school-focused INSET could be profitable. Con- 
ference activities included distribution of papers from previous confer- 
ences, keynote addresses, papers prepared and briefly introduced, group 
discussions around the themes of the conference and the preparation and 
reporting of recommendations for future action. 

Discussions of the evaluation of school-focused inservice education 
and training at the Bournemouth conference were initiated through three 
papers, one by Ray Bolam and Keith Baker (U.K.), one by Harold Eklund 
(Sweden) and one by Kenneth Howey (U.S.). Keith Baker reported on the 
resulting discussions. It pointed 

out, for example, that they were a small group'who had severe diffi- 
culties with terminology. One difficulty pointed out by Baker was an 
unresolved disagreement on the meaning and importance of the term "needs- 
assessment 11 as it applied to the design and eventual assessment of school- 
focused INSET. The group did, however, agree on six principles for 
evaluating school-focused INSET. The following is a critical analysis 
of these six principles. The point of the analysis is to indicate what 
can happen if we limit our view of the context of an inservice education 
and training program to the setting without recognizing the existence 
of different reasons for choosing the inservice education and training 
programs. 

Evaluation is essential to school-focused INSET . This principle 
does not generalize across contexts created by different purposes for 
school-focused inservice education and training. The evaluation of a 
school-focused INSET program is not essential when the purpose is to 
improve school practice. What is essential is an evaluation of school 
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practices and the relationship of these changes to the inservice edu- 
cation and training. The primary concern is the school practice; the 
secondary concern is the design of the program or how the design 
was implemented. 

The essential nature of evaluation to school-focused INSET intend- 
ed to stimulate professional development is a different matter. 
Evaluation of the inservice education activities may be an essential 
part of the instructional process. As a few of the case studies show, 
evaluation may be integrated into the educational activities where in- 
vestigation and reflective analysis are the curriculum (e.g. Olsen, 
Larson, Elliott). When investigation and reflective analysis are the 
curriculum, the evaluation of a school-focused INSET program looks quite 
different from the ways in which most of us have conducted it. There 
are claims being made (e.g. by John Elliott) that if professional dev- 
elopment is both the purpose and the aim of an INSET program, then 
evaluation may be a peculiar and not entirely helpful way to consider 
the relationship between professional reflectivity and professional 
practice. The essential nature of evaluation to INSET for professional 
development may be problematic enough where evaluation may become a less 
solemn and a more personal activity. 

Evidence from the international case studies of school-based INSET 
intended to stimulate professional development and to, improve school 
practice suggests that evaluation may be essential. The reason is found in 
the conflict within a program that is expected to be intrinsic to both the 
individual and to the institution. One solution to resolving the con- 
flict is to show through the evaluation process that the program is re- 
sponsive to the participating educators. Often, unfortunately, other 
conflicting characteristics in school-focused INSET programs intended to 
stimulate professional growth and to improve school practice (see Figure 
1,) are not addressed. There is a lack of vision expressed in 

..these programs for what professional development is or what it can be. 
Related to that is the lack of attention given to resolving theories 
of professional development with theories of institutional change both 
in the design and in the evaluation of these multiple purpose programs. 
Thus, the principle may be a practical response to one conflicting 
characteristic, but it does not illuminate how other conflicting charac- 
teristics are to be addressed when evaluating school-focused INSET in- 
tended to stimulate professional growth and to improve school practice. 

In evaluating school-focused INSET intended to implement social 
policy, what is essential is the investigation of the social policy 
and its implementation; the evaluation of inservice education and train- 
ing is not essential. A primary target in investigating the implementa- 
tion of social policy through school-focused INSET would be the link 
between the school-focused INSET activities and the implementation of 
the policy. The investigation may also need to be comparative, not 
necessarily between inservice education and training strategies, but 
between in< jrvice education and training and other possible strategies 
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for implementing a social policy. This may include cost comparisons. 
In some contexts it may be essential to investigate how combinations 
of purposes (e.g. stimulating professional development and improving the 
school system) can be used to help achieve the aim of the school- 
focused INSET programs intended to implement social policy. Thus, 
stating that evaluation is essential to school-focused INSET is falling 
far short of what would be needed if we were to evaluate a school- 
focused INSET program intended to implement a public policy. Clearly 
theessential nature of evaluation is not in its usefulness to the inser- 
vice education and training activities but rather in its usefulness to 
those who formed the policy and to those who are responsible for seeing 
these policies articulated into school settings. 

In short, the purpose (s) of a school-focused INSET program determine 
whether evaluation is or is not essential. Moreover, the general prin- 
ciple turns attention away from why evaluation is important or what is 
essential to a study of a school-focused INSET program. If school- 
focused INSET, for example, were intended to improve school pxactice, 
certainly essential features of an investigation into that program would 
be on the improvement of school practice and the relationships of the 
INSET activities to those improvements. Too much attention is being 
given by this principle to the inservice education and training activi- 
ties; not enough attention is being given to the differing reasons for 
supporting school-focused INSET. 

Finally, there is an inherent problem to a principle which states 
that evaluation is essential. Making evaluation essential to *school- 
focuscd INSET is one way to reduce, not increase, the possibilities 
that evaluation may bring to the enterprise. It can become a routine 
requirement where the focus may be M did you do an evaluation or not?" 
The reason for evaluating, the focus of the evaluation, its major ques- 
tions, the anticipated problems in school-focused INSET achieving its 
purposes and its aims, the opportunities for a creative response to 
capturing these problems, can become submerged under the essential 
nature of the undertaking. 

The point is not that evaluation is not essential to school- 

focused inset. However, experiences from 

the international case studies suggest it is problematic how an evaluation 
may serve the multiple purposes of school-focused inservice education 
and training programs. The essentiality of evaluation to school-focused 
INSET needs to be addressed according to the aims of the program, whether 
they be to stimulate professional development, to improve school prac- 
tice and/or to implement public policy. 

INSET evaluation is collaborative . The meaning of collaboration 
in an evaluation study differs significantly, given different purposes 
for school-focused INSET. Who may be involved in the evaluation, the 
extent of participation, the process of participation and the integra- 
tion of participation in the program design as well as in the evaluation may 
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be decidedly different, depending upon the purposes for school-focused 
INSET. 

In evaluating school-focused INSET intended to improve school 
practice, the principle has meaning because collaboration is being 
recognized as a viable strategy for getting educators to adapt new 
practices (see McLaughlin and Berman ) , in many of the international 
case studies, for example, collaboration in the evaluation of the pro- 
grams was clearly intended to encourage those who were being changed 
to become part of the change process. The involvement , however , em- 
phasized participating educators 1 analysis of the inservice education 
and training activities; it seldom meant a concentrated effort by the 
program designers, evaluators and participating educators to analyze 
the extent to which school practice was improved (or why). In addition, 
in evaluating school-focused INSET intended to improve school practices, 
there has been a limited view of who are to collaborate. There are 
few risks taken as far as including non-educators, community members 
and those who are normally not included in educational decision-making 
in our respective countries (women and minorities, for example). 
Furthermore, it seems that teachers, in evaluating school-focused INSET 
programs intended to improve school practice, did not readily accept 
their role as collaborators. Some case studies suggested that this may 
have been because the inservice education and training was quite ob- 
viously a means to an end that was seldom made explicit. In some cases, 
of course, it was difficult to address or acknowledge the participating 
educator as an equal partner. In many cases, it would have been inappro- 
priate to assume that practitioners were equal partners, particularly 
when it was their practice that was being improved. 

When the purpose for school-focused INSET is to stimulate profes- 
sional development, collaboration in the evaluation makes sense for 
different reasons. Collaboration between program designer, evaluator 
and participating educators is especially relevant when professional 
development is interpreted as developing powers of self-observation 
and professional reflectivity. In these cases, the medium — evaluation 
and self- investigation — is the message. Evaluation is self-reflective 
inquiry and becomes the aim and process of the inservice education and 
training. In fact, the term collaboration limits rather than describes 
most practices of inservice education and training where the intent is 
to stimulate self-reflective inquiry in participating educators. We 
must remind ourselves, however, that in most cases professional dev- 
elopment has been a purpose but not necessarily the aim of our school- 
focused INSET programs. 

Collaboration in evaluating school-focused INSET intended to stimu- 
late professional development and to improve school practice may have 
double meaning. It makes sense as a method to encourage participating 
educators to change their school practice. It makes sense as a method 
to instill self-reflective inquiry. How both these meanings become 
integrated into one style of collaboration is a problem. Usually the 




style of collaboration emphasizes the need to open participating ed- 
ucators up to specific change*, in their practice rather than to self- 
reflective inquiry into their own assumptions, intentions, and 
practices for their own professional purposes. Unfortunately, given 
the dual purposes, the meaning of collaboration for stimulating pro- 
fessional development tends to be pushed aside. 

When referring to school-focused INSET intended to implement social 
policy, collaboration is obviously more than teachers and program de- 
signers being involved in the evaluation. Collaboration in the evaluation 
of school-focused INSET intended to implement social policy would neces- 
sarily be political. As a principle to follow, it would be ambitious, 
perhaps radical and probably the most significant of all the principles 
stated at the Bournemouth conference. There are some of us, for example, 
who are natural supporters of inservice education and training (e.g. 
university professors, teachers, school administrators). If collaboration 
only included us who support inservice education and training, an analysis 
of the viability of school-focused INSET to implement a social policy may 
not be expected. In practice, there is a strong tendency to reduce rather 
than increase the voices of those who may have a personal stake in the 
social policy but little stake in inservice education and training as the 
means to achieve that policy. Experience shows that collaboration in the 
• evaluation of school-focused INSET intended to implement social policy 
is difficult — perhaps more difficult than collaboration in the evaluation 
of school-focused INSET for professional development or for school im- 
provement. The difficulties are found in finding how the many constitu- 
ents of a social policy can participate in investigating the viability 
of school-focused INSET to implement a social policy. Here it is only 
too obvious that collaboration may become professional versus non-profes- 
sional, enfranchised versus disenfranchised, where collaboration is 
rhetoric used to continue the unequal distribution of influence in 
judging the extent to which social goals are implemented. 

In conclusion, there is probably no principle that is so noteworthy 
and yet so capable of being abused in practice and consequence. Exper- 
ience suggests that specific examples of the meaning, performance, sup- 
port and possible consequence of collaboration will be more useful to 
the profession than an ideal expressed as a single principle. 

Evaluation is important to diagnosis, process study and outcome 
measurement . The significance of this principle changes according to 
the purpose (s) of our school-focused INSET programs. When school- 
focused INSET is intended to improve school practice, diagnosis, process 
study and outcome many be reasonable expectations from an evaluation 
study. Thus, it is interesting, when looking at examples of the inter- 
national case studies, that seldom are the investigators analyzing the 
"outcome": improvement of school practice. Some state that they have 
intentionally stayed away from the analysis of school improvement. Two 
reasons are usually given. One reason is the ambiguous link between 
inservice education and training and school improvement (this was claimed 
even for some programs where a reason for supporting school-focused INSET 
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was to improve school practice). Another reason is a lack of faith in 
the measurement of outcome, particularly in the use of achievement scores. 
Except for some hypothetical models, the case studies suggest that this 
principle is either difficult to follow or reluctantly applied, even 
when the aim of school-focused INSET is to improve school practice. The 
source of the problem is not necessarily the state of the art in measur- 
ing school improvement. There were some case studies, for example, 
which challenged not only the use of achievement scores but even the 
possibilities for analyzing changes in teachers' practice. It is inter- 
esting when the purpose of school-focused INSET is to improve the school 
but the evaluators say that the best that can be done is to describe the 
training process. 

It may be that the principle, especially in the inclusion of 
"outcome measurement 11 , has been found to be most difficult politically 
in precisely those schools that have been targeted for change through 
inservice education and training activities. This suggestion from ex- 
periences of the international case studies is serious for those who may 
wish to see school-focused INSET used to achieve particular improvements 
in school practice. The case studies suggest that evaluation in school- 
focused settings will not be able to indicate to what extent school 
practices were improved, no matter how that improvement is "measured 11 . 
Thus, although the principle seems to make some sense in school-focused 
INSET intended to improve school practice, examples from case study exper- 
ience fall far short of the promise. 

The application of the principle to school-focused INSET intended to 
stimulate professional development is more problematic from the start. 
The distinctions made between evaluation processes such as diagnosis, 
process study and outcome measurement are not relevant to the intent or 
practice of many of these inservice education and training programs. One 
irony in the international case studies was that the problem with this 
principle was not in the ability to interpret appropriate meanings of 
the word "outcome 11 . The few case studies that did have professional 
development as their primary aim were able to articulate quite clearly 
particular features of professional development that were intended to be 
stimulated by the school-focused INSET activities. In fact, the aims of 
those few programs which were intending to stimulate professional develop- 
ment were more clearly stated than the aims of those which were intending 
to improve school practice. Thus, the problem with the principle was not 
that professional development is less able to be "measured" or described 
than school improvement. It is that the distinctions made between diag- 
nosis, process study and outcome measurement as stages of an evaluation 
study conflict with the role and process that evaluation plays in those 
inservice education and training programs whose aim is to stimulate 
professional growth. 

In school-focused INSET intended to implement social policy, the 
distinction between diagnosis, process study and outcome measurement 
as stages of evaluative study may be appropriate but they are not neces- 
sarily the most important features of an evaluation study. The focus on 
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the implementation of social policy (the "outcome") is essential. 
Diagnosis may or may not be important to evaluating school-focused 
INSET intended to implement social policy. Usually, for example, 
social policy has already been formed based on some previous indica- 
tion of general need. 

The problem with this principle, however, is that it does not 
refer to some of the most essential characteristics of an evaluation 
of school-focused INSET intended to implement social policy. An eval- 
uation of INSET designed to implement social policy, for example, would 
necessarily include comparisons with other means for achieving the poli- 
cy. ' Cost-effective comparisons or cost-benefit studies may be required. 
For social policy, a process/outcome evaluation is not enough. An an- 
alysis of the outcome reached and the extent to which the process used 
is linked to that outcome is useful but it is less than a policy maker 
needs. The policy maker usually needs more information on the related 
benefits of the possible alternatives to the school-focused INSET 
strategy being used. Again, one basic concern of the policy maker 
is that INSET is only one way among many in which a particular social 
policy may be reached. 

If we look at the few studies that were evaluations of INSET for 
social policy, they had similar characteristics to those which evalua- 
ted INSET for system improvement. There was an emphasis on the INSET 
used and its acceptance or rejection by the recipients, but compara- 
tively little attention given to the outcome, the policy aim. When 
the outcomes were addressed directly, the possible alternatives to 
using INSET as a means or the relative merits of different means to achieve 
similar social policies were not addressed. The principle, then, may 
seem to apply to the evaluation of school-focused INSET intended to 
implement social policy, but perhaps detrimentally to the investigatory 
and advisory responsibilities of the evaluation studies. 

INSET evaluation is part of the teachers' professional responsibility . 
Teacher professional responsibility for evaluating school-focused INSET 
loses meaning and relevance depending upon the purposes of the inservice 
education and training. When school-focused INSET is intended to improve 
school practice, the principle offers an alternative to professional 
reward: make it a job responsibilty . Experience is suggesting, however, 
that it may be more useful to improve school practice if the teacher were to 
be partly responsible for evaluating the extent to which the school has 
been improved rather than evaluate the process of the school-focused 
INSET program. An emphasis on teachers evaluating the inservice educa- 
tion and training is misplaced. Instead, if teachers analyze the in- 
fluences of specific inservice education and training processes on their 
own practices, it may inadvertently improve their practice. 
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When stimulating professional development is the aim of school- 
focused INSET, the principle may be more applicable but less creative. 
As the one for whom the school-focused INSET is designed, part of a 
teacher's responsibility is to evaluate the extent to which the experi- 
ence is enriching and personally satisfying. To this the principle adds 
little. A far more risky statement, but one with nearly the same mean- 
ing when applied to school-focused INSET intended to stimulate profes- 
sional development, would be to state that professional development is 
part of a teacher's professional responsibility . 

When applied to school-focused INSET intended to implement social 
policy, the principle is exciting and limiting. The excitement is 
that teachers are seldom, if ever, responsible for evaluating the ex- 
tent to which a social policy is implemented (and why). The limitation 
is that teachers are not the only actors in school-focused INSET intended 
to implement social policy tc whom this principle may be significantly 
applied. 

Thus, the principle may be creative and exciting if it is applied 
to school-focused INSET intended to improve school practice or to im- 
plement social policy only if the responsibility focuses on the evalu- 
ation of school improvement or on the implementation of social policy. 
In school-focused INSET intended to stimulate professional development, 
the principle is redundant. Instead of this principle, more specific 
advice is needed on the resources and policies for supporting particular 
levels of teacher involvement in the evaluation of those school-focused 
INSET programs intended to improve schools or to implement social policy. 

INSET evaluation is required at many levels within schools as 
organizations . This principle is the most generally applicable of the 
six formed at the Bournemouth conference. Emphasizing schools as or- 
ganizations is particularly relevant to school-focused INSET intended 
to improve school practice. The school is organizational, has many 
levels of participants, is hierarchical and operates in routine proced- 
ural ways. School improvement may necessarily include changes in the 
organizational structure; changes in the routine or practice of one 
level in the school may be contingent upon the capacity of another 
level to change. It is reasonable to suggest that evaluations of school- 
focused INSET intended to improve school practice direct their investi- 
gations to the organizational systems of the school in which the inser- 
vice education and training is performed. Many of the international 
case studies said as much in their closing statements. 

The many leveled features of schools as organizations provide a 
different set of problems to the evaluation of school-focused INSET 
intended to stimulate professional development. When professional 
development is the aim of a school-focused INSET program, the organiza- 
tional context in which the education and training occurs and in which 
the development may take place is often considered as extraneous noise 
and bothersome reality. On the other hand, the few case studies of 
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programs where professional development was the aim indicated that 

.pportive resources of the organization were necessary if professional 
development were to be significantly initiated, maintained or encouraged 
to grow. Unfortunately, these observations were usually made at the 
end of the investigations and were not a part of the focus of the evalu- 
ation studies. The collective adv£c'£',* however, is clear: if profession- 
al development is the aim of a school-focused INSET program, it would 
be useful to direct some attention to the organizational context (see 
Anglin et. al; Meyer). 

The organizational context is particularly interesting when school- 
focused INSET is intended to implement social policy. Experience from 
the international case studies as well as other studies (see Corwin; 
Fox) of evaluations of inservice education and training programs intended 
to implement social policy suggest the crucial importance of the organ- 
izational contexts. Furthermore, experience shows how essential it is 
to analyze the extent to which a social policy is implemented from a 
perspective of what is organizationally possible. Studies that have 
investigated school-focused INSET intended to implement social policy 
take the school organization as being problematic, that is, as an in- 
fluencing factor that can be changed. Investigations into school-focused 
INSET intended to improve schools or to stimulate professional develop- 
ment can do likewise. 

Experience suggests that evaluations of school-focused INSET can 
benefit from an investigation of how the inservice education and train- 
ing is influenced by, and influences, the many levels of school organi- 
zations. A major problem pointed out in the case studies, however, is 
not in the significance of the principle but in how the principle may 
be applied. The concern is how the organizational context can be criti- 
cally examined in school-focused INSET programs intended to improve 
school practices or to stimulate professional development. It is a 
natural consideration in evaluating school-focused INSET intended to 
implement social policy. 

INSET evaluation should be illuminative * In beginning our analysis 
of the general applicability of illuminative evaluation for different 
contexts of school-focused INSET programs, it may be well to remind our- 
selves how Parlett and Hamilton described illuminative evaluation and 
the context in which they presented their argument. "Illuminative 
evaluation" was directed towards the evaluation of innovative curriculum 
programs — of designed interventions upon schooling practices. It was 
directed at one purpose of inservice education and training: to improve 
school practice. In focusing on innovative curricular interventions to 
improve the schools, Parlett and Hamilton described in quite explicit 
detail what they meant by the terra "illuminative evaluation". The 
reports of an illuminative evaluation, for example, should make a 
program recognizable to others outside the innovation, where the conclu- 
sions may increase rather than lessen the sense of uncertainty, where 
theoretical principles underlying the investigation are made explicit 
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and where evidence is presented in such a way that others can judge its 
quality and make alternative interpretations. They also articulated 
particular views of what is an educational program (e.g. a unique 
pattern of circumstances) and what is the purpose for evaluation (to 
understand the innovative process and its intended and unintended 
consequences) . 

Parlett and Hamilton were addressing the evaluation of those situ- 
ations where curriculum programs are designed (usually outside of the 
school setting) and then placed into the schools. Their emphasis on 
process} reporting procedure, curriculum theory, ambiguity and dialogue 
assumed a situation in which school practice was being improved from 
external professional sources. Their argument was that in evaluating 
curriculum innovations, the professional respect for the educational 
milieu of curricular programs (the teachers, students, existing curric- 
ulum) and especially the intentions of all actors needed to be 
considered. The power of their argument was in the view that theoretical 
perspectives and instruct ional/curricular intentions were held by all 
actors in the innovative enterprise and that these perspectives, atti- 
tudes and responses (rational and emotional) to the innovation were 
significant factors in the performance and eventual successes and fail- 
ures of particular innovative curriculum programs. 

How does illuminative evaluation articulated for evaluating innova- 
tive curriculum programs become adapted to the evaluation of specific 
school-focused inservice education and training programs? The principle 
does not make that clear. If we look closer at the three purposes that 
can be held for school-focused INSET, it is apparent that the adaptation 
would be quite different depending upon the context created by our in- 
tentions for inservice education and training. 

In applying a perspective of illuminative evaluation to the eval- 
uation of school-focused INSET intended to improve school practice, we 
have two choices. We can interpret the "curriculum innovation 11 to 
be the practices we intend to instill in the school. With this inter- 
pretation of the innovative feature in an INSET program, applying an 
illuminative evaluation would be comparatively straightforward. We 
would, as Parlett and Hamilton suggest, focus on the instructional 
milieu of the school, participants 1 theories of instructional practices, 
their perspectives on what is occurring in the INSET and the attitudes 
and responses to what occurs. The inservice education and training 
program would be only a part of the focus of the evaluation.. It would 
be subsumed in a larger perspective of curriculum, one that was particu- 
larly related to the practices being intervened upon by the program. 
As rich a challenge as this interpretation brings to an illuminative 
evaluation of school-focused INSET intended to improve school practice, 
no international case studies applied an illuminative perspective in 
this way. Instead, they interpreted the curriculum innovation to be 
the inservice education and training program itself. This choice 
resulted in placing the focus on the theories of teacher education and 
curriculum, on what occurred in the school-focused INSET activities 
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and its impact on the participants. Illumination was sought for teacher 
education rather than for school change. Either choice is possible in 
applying the principle of illuminative evaluation to school-focused INSET 
intended to improve school practice. The latter, however, overinterprets 
the degree to which participating educators have a theory of teacher edu- 
cation. Stating that an evaluation of school-focused INSET intended to 
improve school practice is to be zn illuminative evaluation does not 
clarify what will be done, nor even what will be interpreted as the 
intervention. 



Applying an "illuminative" perspective to the evaluations of school- 
focused INSET intended to stimulate professional development is no prob- 
lem. Clearly the innovation is the INSET program. Thus, most of the 
features of the illuminative perspective described by Parlett and Hamil- 
ton are consistent with most aims for professional development. Implicit 
theories of professional development, for example, are held by all par- 
ticipating educators, what occurs in the inservice education and training 
and its intended and unintended consequences are unquestionably relevant 
to the intent and purpose. of the school-focused INSET prograns. In circum- 
stances when school-focused INSET is intended to stimulate professional 
development, the critical views to understanding the program are those 
views about teacher education. The few international case studies that 
focused on programs aimed at stimulating professional development show 
how an illuminative perspective can be readily applied to evaluating 
these inservice education and training programs. 

Apparent from other studies, however, is the problem -in trying to 
apply an illuminative perspective to INSET programs that are intending 
to improve school practice as well as to stimulate professional devel- 
opment. In evaluating these programs, the unfortunate tendency has been ~ 
to focus only on the illumination of the inservice education and train- 
ing and its stimulation of professional development to the detriment 
of the added purpose: to improve school practice. To adapt an illum- 
inative perspective to school-focused INSET with multiple purposes is 
a very difficult undertaking because it places the program into more 
than one context. Each context has a different theoretical base; each 
context has differing spheres of influence and consequence. 

An illuminative evaluation of school-focused INSET intended to im- 
plement social policy is interesting to imagine. Clearly it would place 
the inservice education and training in a mileu that is wider than the 
school or than the field of teacher education. The implicit theories 
held by participants may be more difficult to ascertain as well as more 
couched in everyday language. Moreover, the range of participants would 
be greater, including, of course, the policy makers and the intended 
beneficiaries of the social policy. The challenge is interesting but 
the principle gives no insight into how an illuminative investigation of 
a school- focused INSET program intended to implement social policy would 
be designed or carried out. (There is one study currently being tried 
in the U.S., a three to five year study of the Teacher Corps program 
that is trying to adapt an illuminative perspective. The difficulties, 
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particularly in terms of illuminating social policy, have 
been enormous , jr- " 

Thus, as attractive as the illuminative perspective 
is to many of us policy makers, program designers and 
evalua-cors, the principle is not nearly as helpful as we 
would like - or need. -It is not apparent from the principle 
how the notions of illuminative evaluation will be applied 
to mservice education and training programs aimed at improv- 
ing school practice or at implementing social policy. 
Particularly for those two purposes of school-focused INSET, 
there may be far more groundwork, preparatory dialogue 
and maintenance needed to initiate," implement and sustain 
an illuminative evaluation of school-focused INSET programs 
Jian is suggested by the stated principle. 
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C. Conclusion and an Optimistic Solution 

The six principles formed in the Bournemouth conference 
indicate a refreshing sensitivity to the adult world of 
inservice education and training and an uncommon zeal for 
insightful evaluation. Hov/ever, they may restrict the 
creativity of evaluation and reduce its power to inform. 
Even though the settings were similar (school-focused INSET), 
the principles formed at Bournemouth take on different meanings 
and levels of significance depending upon our purposes for 
inservice education and training. YJhat is needed ar.e more 
detailed guidelines for specific contexts and more direct 
attention to specific problems affiliated with particular 
contexts . 

If we begin to look at the purposes of school-focused INSET, where 
do problems in evaluation occur? This analysis of the international 
case studies suggests that one obvious situation fretted with problems 
in evaluating inservice education and training is when more than one 
purpose is intended. Most of these principles, for example, are dif- 
ficult to interpret or apply in programs that intend to improve school 
practice and to stimulate professional development- That does not 
necessarily imply that we should not or cannot have more than one 
purpose for our inservice education and training programs (or have 
multiple purposes and aims). In fact some of the most interesting 
programs evaluated and some of the most creative approaches to evalua- 
tion are found in three international case studies where multiple 
purposes were intended. It does suggest, however, the seriousness 
of the rhetoric we use as reasons for supporting an inservice education 
and training program. Our rhetoric creates part of our context and we 
may be creating a context that is more complex than necessary. 

Another conclusion is implied as we consider inservice education 
and training intended to implement social policy. The analysis of 
the principles formed at Bournemouth indicates just how much work needs 
to be done if we are to evaluate our programs in this context. There 
are relatively few examples of evaluations of inservice education and 
training programs intended to implement social policy. In addition to 
there being little experience, the analysis suggests just how different 
an evaluation in this context would be from evaluations of programs 
intended to improve school practice or to stimulate professional devel- 
opment. Theories of social reform, the perspectives used for analysis, 
the focus on the policy and the policy making process, the embeddedness 
of educational practice in our respective societies become central to 
evaluations of inservice education and training intended to implement 
social policy. There are some of us who suggest that all programs are 
intended to implement social policy. If that is our intention, then 
our evaluations of inservice education and training would necessarily 
be conducted with far more intention to social policy, its formation 
and its context than we have given in most of our past studies of inser- 
vice education and training. 
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Our. intentions for inservice education and training can be just 
as significant in ad-hoc groups and multiple school settings as well 
as in school-focused settings. The setting, after all, is part 
of the strategy used to reach our intentions for inservice education 
and training. Whether we intend to stimulate professional development, 
to improve school practice or to implement social policy, there is no 
foregone conclusion which setting may be most viable. The choice of 
setting does not imply purpose. Thus, we need to analyze our actions 
and learn our lessons not only in light of the settings we perform in 
but also in light of the intentions we hold. 

The conclusion is simple. If our basic intent is to improve 
school practice, say so; if it is to stimulate professional development, 
say so; if it is to implement social policy, say so. If we want to 
combine more than one purpose for our inservice education and training, 
do so with the knowledge that we are creating a* significant challenge 
to evaluating these programs. The worst possible situation, however, 
is one that has happened too often in these international case studies: 
that is when we take our rhetoric more seriously than we intended. 
Too often, for example, "professional development" became a slogan we 
used to gain support of an inservice education and training program 
that was intended for quite a 'different purpbse. That is not necessar- 
ily all bad but then we evaluate the program as if it were primarily 
intended to stimulate professional development . If we are more careful 
about the rhetoric we use in supporting inservice education and training 
programs and less inclusive in some of our stated intentions, perhaps our 
inservice education and training programs can be more effective. Cer- 
tainly the evaluations of these programs would be more realistic and 
perhaps eventually more illuminative to us policy makers, program de- 
signers and evaluators. 

The solution? Articulate the reasons \re hold for 

supporting inservice education and training* Consider _ 
the setting (e^g. ' school-i pcusedT as" one strategy 
for meeting our purpose(s) for inservice education and training. 
Analyze our experience in terms of the context we have constructed by 
our intentions for performing inservice education and training and by 
the chosen setting, and then determine what we have to do to evaluate 
our programs better given our intentions. To disregard our intentions 
for inservice education and training is to limit reflection upon our 
experience and to impair our ability to do better next time. 
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III. Issues to Confront When Reflecting Upon The 

Evaluation of Inservice Education and Training 



A. Confronting Mundane Realities to Evaluating 
Inservice Education and Training 



Further reflection on evaluating inservice education and training 
needs more than has been provided thus far. The six principles from 
the Bournemouth conference serve only as a starting point for dialogue. 
The international case studies are too tame and too cautious to stand 
alone as primary sources for candid discussion. Future references to 
our intentions for inservice education and training (to stimulate 
professional development, to improve school practice, to implement 
social policy) are likely to be hollow if they do not refer to the 
irascible and the difficult in our performance. What is needed is to 
place our past performance into a realistic and grounded perspective. 
If we are to use our past performance in INSET evaluation for redevel- 
oping future performance, a way is needed to bring up for public inspec- 
tion the challenging and peculiarly exasperating features of what we 
do when we evaluate inservice education and training. 

This is not as easy as it may sound. Telling stories of our past 
performance, for example, is not enough. As we tell our stories, we 
are sure to M make sense and lose meaning" (as Walker reminds us), not 
because we necessarily embellish our performance but just the opposite, 
we necessarily leave out those portions of what we did that do not make 
sense. In recreating our actions, our understanding gets in the way of 
our experience; we reduce the range of recollection to the sense of the 
story that we can tell others. There are ways to urge experience to be 
recalled. One way is to create fictional accounts that place disparate 
features of experience together in order to stimulate new understanding. 
(Walker tries this in a 1979 NSF case study e ) There are other ways. 
Many are creative in their approach to inducing recollections, begging 
for reconstructions of personal knowledge. Simulations, photography, 
music, poetry, highly personal vignettes and directed challenges from 
"foreign 11 perspectives (e.g. teacher vs. student) all have been used 
to stimulate us to bring up for inspection previously unnoticed features 
in our work. In a recent OECD conference, there was a more common ' 
method used to stimulate the recall of the peculiar, challenging and 
surprising aspects of our work in evaluating inservice education and 
training. That method was the raising of issues from personal experi- 



Experience began to surface in a two day meeting between the writers 
of the international case studies on the evaluation of inservice educa- 
tion and training, managers of OECD, the reporter of the Bournemouth 
conference and this writer. The issues were formed, related and discussed 
in terms of personal professional experience. Questions were raised. 
Problems were exemplified. Old issues were redirected by participants' 
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more recent experiences in evaluating inservice education and training. 
All participants had performed INSET evaluations themselves. Most had 
written reviews of some of their compatriots' efforts and felt comfort- 
able that they could represent more than their own views on evaluating 
inservice education and training. Most had attended previous OECD 
sponsored conferences on inservice education and training; 
some had arrived at the working principles at Bournemouth* 
The resulting two days of discussion, then, could be considered as an 
extension of the experience begun by the writing of the international 
case studies and the discussions of previous OECD conferences on 
evaluating inservice education and training. . Issues were raised from 
members 1 direct experience gained in evaluating inservice education 
and training as well as from vicarious experience gained from reviewing 
others' evaluations. Members had been policy makers and program 
designers as well as evaluators in their respective countries. 

Of course the two day conference must also be judged and understood 
as an international two day meeting directed towards raising significant 
issues. Ten individuals from different countries, holding similar 
professional concerns and building parallel careers, were expected to 
reconsider their experience and open their views for analysis in a 
spirit of professional dialogue. We know this seldom happens whether it 
is in a national or an international context. In such settings, the 
natural process is first to carve out one's own professional image, then, 
if time is left, to engage in some dialogue from the positions that have 
been created. This process occurred and professional images were formed 
before much dialogue began and were sustained as issues were being 
raised. I have tried to capture the irascible problems that the ten 
members at the conference considered significant as they recalled their 
experience in performing and reviewing evaluation of inservice education 
and training in their respective countries. 

It is important to realize that the procedural intent to raise 
issues from experience was different from previous OECD conferences on 
inservice education and training. The intent was not to arrive at 
principles for evaluating inservice education and training, nor was it 
to come to a consensus on what made good evaluations. Instead, the 
intent was to raise what from their own experience seemed to make 
evaluating inservice education and training challenging, frustrating 
and difficult. Although there was a lack of reference to the intentions 
of inservice education and training, there were occasional outbursts 
requesting more clarity of the special problems faced in evaluating an 
inservice education and training program that is intended to stimulate 
professional development and to improve school practice. That concern 
was understood but not addressed. 

What was addressed were three mundane realities of evaluating 
inservice education and training. These realities are: 
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1) 



the bureaucratic context of evaluating inservice 
education and training; 



2) 



the choice of methodology for evaluating 
inservice education and training; 



3) 



the many meanings of "participation" when 
evaluating inservice education and training. 



Although the phrase "mundane realities" may be redundant (mundane 
being worldly), it is meant to denote our understanding that a) we all 
know that these features of our work exist (that is, they are part of 
our world in evaluating inservice education and training) and b) they 
seem to be distinct where we tend to float in and out of them at will 
but seldom stay in two at once (thus "realities"). Most of the issues 
raised emanated from these realities. 

The intent of the following is to point to the rich fabric of our 
experience. Considering the profane nature of these realities and 
the limited representation of ten persons from six different countries, 
the issues are dense, lively and worthy of our consideration. They may 
help those of us who have past experience in evaluating inservice 
education and training to recall significant features of that experi- 
ence that we may have forgotten. They may help those of us who have 
little experience to anticipate what may be significant if we become 
involved in evaluating inservice education and training. The variety 
of issues is not exhaustive, nor are the three realities. The following 
issues were raised as lessons learned from individual experience. They 
were not generalizations, they were not intended to be corraborated 
through consensus. Instead, they were to be understood as reflective 
observations formed in limited contexts. The significance of these 
observations to other specific contexts is expected to vary and to be 
established only through analyses from other experiences (such as those 
of the reader). 



B. Issues Proceeding From The Bureaucratic Context 



Bureaucracy is not used here in a pernicious sense. A modern 
bureaucracy is staffed by persons who have been educated to a degree 
not anticipated by most critics of bureaucracies. Simple stereotypes 
do not hold. Ignorance, misunderstanding and limited perspectives 
are no more descriptive of the modern bureaucrat than they are of the 
evaluator or teacher educator. In education, of course, almost all 
of us are bureaucrats, funded by public monies to perform public 
services. Institutional routine and regimented rules of procedure are 
well understood by all of us, no matter what our positions in our 
respective educational agencies institutions • 
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Bureaucracies are a nearly universal phenomenom in modern Western 
cultures. They exist as practical agencies to deliver public services. 
They are to ensure that policies made by our elected officials are im- 
plemented. The difference between the policy as intended, as written 
in legislation may become quite distinct from the policy as interpreted 
by the bureaucrat (see Edelman). This is where bureaucratic power may 
be over interpreted by either the bureaucracy, the client, or, as is 
sometimes the case, those who are contracted by the bureaucracy to 
perform a service (like evaluation). They are, however, interpretive 
bodies with a range of decision-making responsibilities that may go 
beyond their authority. They are usually hierarchically organized with 
distinct levels of responsibilities formally maintained. This hierarchy 
is often compressed into a horizontal view of the agency by those who 
are receiving the direct services (see Lipsky). The modern bureaucrat, 
then, is comparatively well educated, may have some flexibility in 
interpreting policy and making decisions, exists in a hierarchical 
organization but is considered by the clients to be singularly 
responsible and authoritative. In short, the context in which a modern 
bureaucrat works is very complex. 

In the complex context of the bureaucrat, a view towards science 
as positivistic, empirical inquiry, capable of universal acceptance and 
recognized as legitimate sources for particular decisions can be under- 
stood. The need of most bureaucracies is to reduce their complexities, 
making their decisions appear more rational and reasonable to internal 
critics as well as to external clients. 

Inservice education and training is almost always embedded in a 
modern bureaucracy. It is and has been very often designed as the 
means through which an educational policy shall be reached or a social 
aim may be implemented. Regardless of the setting or who initiated the 
inservice education and training, its evaluation is nearly always 
sponsored by a bureaucratic, governmental agency (local, regional or 
national). This mundane reality that a bureaucratic context exists 
raises a number of issues about the performance, perspective, purpose 
and audience for the evaluation of inservice education and training. 
The following are only some of the issues which were expressed at the 
international conference on INSET evaluation by those who had performed 
and analyzed evaluations of inservice education and training in their 
respective countries. 

Bl. There may be nothing as "political 11 to a bureaucrat as the 
evaluation of inservice education and training . A crucial difference 
between the bureaucracy which interprets public policy into action and 
the legislature which creates public policy is the meaning of "political". 
To an elected official, "political" may refer to party affiliations and 
active electoral constituencies. The political process of collaboration, 
involvement and negotiation has everything to do with representative 
democracy and little do do with constituencies 1 professional status. 
In a bureaucracy, "political" may have a quite different meaning. 
Collaboration and negotiation may first relate to dealing with the internal 
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relationships which make up much of the office reality of a bureaucrat. 
Furthermore, the political process in a bureaucracy is focused on how 
to deal with professionals who represent conflicting but powerful 
interests. In evaluating inservice education and training, a variety 
of conflicting professional interests may be affected and many bureau- 
cratic levels may be involved. Thus, from a bureaucratic viewpoint, 
the evaluation of inservice education and training is a highly political 
activity. To an elected official, however, the evaluation of inservice 
education and training may not be political because of the relatively 
small constituency directly involved. 

B2 * There is a respectability and a legitimacy to pseudo-scientific 
results. Bureaucratic sponsorship of evaluations of inservice education 
and training need objective accounts and judgments that can reduce the 
amount of conflicting professional opinions and can increase the legiti- 
macy of their own interpretations of what is appropriate action in im- 
plementing public policy. Since there is no science that can provide 
such certainty, a pseudo-science is nearly as effective. Results that 
look objective and free of bias, results that look like they must be 
universally accepted and cannot be challenged are what a bureaucrat 
can use most efficiently. The efficacy and desirability of a needs- 
assessment and of a process-product approach to evaluating inservice 
education and training are examples of pseudo-sciences. This respect 
for pseudo-science may also be supported by some members of the academic 
community who hold narrow professional views of what "science" is when 
applied to the investigation of educational programs. Phenomenological 
views of science such as those of Toulmin orFeyerabend are seldom enter- 
tained. 

B3. Information from evaluations of inservice education and 
training may be minimally related to bureaucratic policy making procedures . 
Policies may be made at a legislative level for one set of purposes for 
supporting inservice education and training and interpreted at various 
bureaucratic levels for quite different purposes. This causes problems. 
One problem is who is the audience (it is usually the bureaucracy but 
not always). A second is what kind of information could be relevant 
to particular decisions. For whom are the reports designed? Not only 
may a bureaucrat's need for information be different from a legislator's 
need but different bureaucratic levels may require differing kinds of 
information. The fragmentation of decision-making may itself have 
serious implications to the conduct of the study. It has been noted by 
many investigators of policy making that evaluative information may have 
little relation to specif ic* policy making procedures (see Lindbloom 
and Cohen, Weiss). Experience in evaluation suggests a more problematic 
relationship between evaluative information and policy than has been 
articulated in the literature. 

BA. The bureaucratic context may determine some or che most crucial 
problems faced in the reporting of results . One view of the international 
case studies, shared by the writers and readers on the reports, is that 
they seem to be written from a singular context. The cultural differences 
are not readily apparent, a singular culture of Western bureaucracy is 
implied. It is the bureaucratic political milieu wliich may make these 
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reports much more mundane than the actual experiences. In addition, 
the fragmentation of responsibility, the many levels of decision 
making that are part of the bureaucratic organization make reporting 
from INSET evaluations either a nightmare or impossible. Even within 
the sponsoring bureaucracy a singular report is inappropriate. The 
most serious issues, however, are those that ask how more candor, hon- 
esty, humor, pathos and passion can be included in the evaluation re- 
ports of inservice education and training. One would think that in- 
vestigations into the work of adults educating adults may be particular- 
ly capable of producing creative reports that touch many of our personal 
and professional nerves. We may need to be careful in selecting how we 
portray children but need we be so timid about our own portrayals? 
(see MacDonald) The claim here is that many of the inadequacies in 
what we report emanate from the bureaucratic context of INSET evaluation. 
If our reports are to be more. adequate and lively, they will have to 
be made so within their bureaucratic contexts. 

B5. We who evaluate have our own bureaucratic context . Increased 
tendencies towards specialization and certified professionalism of the 
evaluator suggest that the bureaucratic context of the evaluator may be 
significant to evaluation of inservice education and training. The 
choice of evaluation design or the nature of the questions to be asked 
may be determined by the evaluator *s past and place in a bureaucratic 
hierarchy. The flexibility of evaluators to respond creatively to 
challenges in evaluating inservice education and training may depend 
upon the evaluators 1 abilities to manipulate their own organizational 
contexts. 

B6. A theory-practice dichotomy is reinforced, if not created, by 
the bureaucratic context . From a bureaucratic viewpoint, a role respon- 
sible for producing theory about an educational practice is considered 
to be higher than the role performing the practice. If A is theorizing 
about an educational practice (e.g. teaching mathematics), A is nearly 
always in a higher and different position than the person performing 
the practice. Hierarchical distinctions between practitioners and 
theoreticians result in unnecessary implications. A hierarchical dis- 
tinction suggests that practitioners 1 theories about their own practices 
are insufficient for investigation. What is worse, it suggests that the 
practice of theoreticians may be too difficult to pursue, even when it 
may be particularly relevant to an evaluation of inservice education 
and training (e.g. who is to evaluate the professional development 
of professional developers?) Another implication is that a person lower 
in the hierarchy cannot be an effective evaluator of functions performed 
by those higher in the system. The reason given is that they do not 
have theories of these functions. Teachers, for example, do not have 
theories of development or theories of school improvement or theories 
of social reform that would make them appropriate investigators. Thus, 
the question of who should evaluate inservice education and training 
and what should be investigated is often limited by the theoretician- 
practitioner dichotomy supported by bureaucracies involved in the in- 
vestigation. These problems are acute whether it is professional devel- 
opment, improved school practice or the implementation of social policy 
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that are being evaluated. 



B7. The bureaucratic context is particularly significant when 
evaluati ng the role of inservice education and training in the redis- 
tributio n or maintenance of power . When the intent of the inservice 
education and training is to implement social policy (for example, to 
redistribute educational opportunity), it is necessary to penetrate 
and analyze the bureaucratic context itself. The reason is obvious. 
The bureaucratic context may be a significant determinant of the suc- 
cesses and failures of these socially oriented programs. This would 
include an investigation into the bureaucratic context of the client 
agency, the sponsoring agency and the evaluators, all of which are not 
usually performed. Some of the most penetrating perspectives that may 
be used to investigate the role of the bureaucratic context in the 
implementation of social policy have political ideologies that look 
risky in the bureaucratic context. Analyzing inservice education and 
training as a process of cultural reproduction (e.g. Lundgren and 
Petterssen; Bernstein) or as the unequal distribution of economic 
opportunity (e.g. Apple) may be particularly insightful but are seldom 
considered. Even MacDonald with his emphasis on democratic ideals is 
seldom referenced. In cases where inservice education and training is 
performed to implement social policy, ideology as a personal construct 
of political meaning may be significantly related to what is and what 
is not done in these programs. 

In summary, the bureaucratic context of the evaluation of inservice 
education and training may be the source of a variety of crucial issues 
that need be addressed by policy makers and designers as well as by 
evaluators. Many of the most difficult questions about evaluating 
inservice education and training arise from its bureaucratic context and 
must be addressed as they relate to that context. Further understanding 
of the difficulties in implementing and assessing our performance in 
evaluating inservice education and training will require reflective 
inquiry by all involved parties into how the reality of their own 
bureaucratic context impinges upon the intent, design, conduct, communi- 
cation and consequences of their investigations. One problem is getting 
persons who u o not normally consider themselves as bureaucrats (e*g. 
the evaluators, the trainers, the designers) to understand their own 
bureaucratic context. What is important to consider is that the 
bureaucratic context is a self-created reality. Those features of this 
reality that we think are least desirable or too influential on certain 
issues can be changed. It is not only those issues that emanate from 
the bureaucratic context that may need to be addressed as we evaluate 
inservice education and training, it may be the source itself. 



C. Issues Proceeding From the Choice of Methodology 



A second reality from which emanate a variety of issues on the 
evaluation of inservice education and training is the choice of 
methodology for the evaluation. The reality is the need to choose 
a method for- investigation. The methodology used is^not automatic, it 
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is not determined by context; it is determined by us. Thus, the method 
used to evaluate a particular inservice education and training program 
may be eclectic and heuristic on rational grounds. The choice of 
investigatory method may also, of course, be influenced by the bureau- 
cratic context, the bureaucratic contexts of the sponsoring (and client) 
agencies and of the evaluators. A legitimate reason for selecting 
one method of investigation over another, for example, is the nature 
of information that is considered to be most related to the view of 
the inservice education and training policy makers (e.g. middle level 
managers in an agency). Another is what the bureaucratic context of 
the evaluator will allow. 

The issues that emanate from the reality of having to choose a 
particular evaluation methodology refer to what designs and procedures 
are available and in what circumstances they have proven useful or not 
useful. They may also refer to the process used to choose a methodology. 
It may be necessary to note that in evaluating inservice education and 
training these issues are not only problems to the the evaluator but 
to the policy makers and designers of inservice education and training. 
The following are some of the issues which were raised by those who 
had recently performed and analyzed INSET evaluations in their respec- 
tive countries. 

CI* Case study, as a focus on particular instances, has inherent 
strengths and weaknesses for evaluating inservice education and train- 
ing programs . One strength of case study is the flexibility in tradi- 
tional uses. It can mean naturalistic portrayals, theoretically in- 
spired investigations or historical analysis and aggregations. Each 
has a strong tradition., is grounded in a different set of underlying -;j£v 
assumptions and may be used in quite different contexts or circumstances, 
Naturalistic portrayals are stories told to capture what happened and 
what it felt like to the participants. The challenge is to tell it 
like it is, like it was felt to be. It is the imaginative recreation 
of natural events. Robert Stake, Barry MacDonald and Stephen Kemmis 
have provided significant examples of and rationales for naturalistic 
portrayals. Theories, if there are any, are implicitly contained in 
the stories and are open to multiple interpretations by the readers of 
naturalistic portrayals. Theoretically inspired investigations, on 
the other hand, are focused on the formulation and/or analysis of 
particular theories. The challenge is to make the theory explicit and 
grounded in the reality of specific circumstances. Glaser and Strauss ^ 
describe how grounded theory may be generated through case study; Parletfc*; 
and Hamilton suggest how case studies can be used to illuminate the 
relative successes and failures of competing educational theories. Sten- 
house describes the historically inspired case study and the aggregated 
forms of case study: the historical survey. The challenge of the 
historical case study is to capture particular decisions and .their 
consequences in light of specific contingencies. In evaluating inser- 
vice education and training, capturing what happened, illuminating the 
underlying theories of why it happened and relating how decisions are 
made in light of contingencies all make a great deal of sense. One issue 
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is which tradition to use. If more than one tradition is appropriate, 
can they be used simultaneously? 



Many other issues emanate from the intent to focus on particular 
cases. The aggregation of case studies is still a problem (not solved, 
for example, by Stake and Easley). So,, too, is the acceptability or 
legitimacy of case study to decision makers (e.g. the bureaucratic 
policy maker or the elected officials who have chosen inservice educa- 
tion and training for a reason). This is becoming less a problem as 
more examples of how case studies can be applied to policy decisions 
are produced (e.g. MacDonald, Rist). I have been told, for example, 
that in the U.S., the reviewers of a proposal for an investigation now 
going on (Bussis, Chittenden and Amarel) had no trouble with the case 
study features of the study design but they had, with the same study 
design, an almost fatal aversion to the participatory features of the 
design. Since case studies can not be considered to b.e representative 
instances of a particular class, the acceptance of case studies suggests 
that it is the unique situation that is being accepted by bureaucratic 
policy makers. In the context of specialization of responsibilities, 
it may be the unique case study, both naturalistic, and theoretically 
inspired, that is most valuable. For a middle-level policy maker., 
circumstance, contingency, underlying assumptions and consequence may 
be remarkable in their power to provide insights from a single case. 

Generalisation becomes an issue to case study, sometimes as a 
result of the misinterpretations of what can be generalized from 
quantitative studies (see Levin, Hamilton). The distinctions made 
between generalization to (from instance A to instance B) and gener- 
alization from (from a collection of instances to a generalized case) 
are helpful in describing how case studies may be considered to be 
generalizable (see Hamilton). Also helpful are the stated intentions 
of naturalistic studies (where generalization is the responsibility 
of the audience) and grounded theory case studies (where a comparison 
and contrasting of instances helps develop hypotheses). 

Some maintain that the case study method is too reflective for 
the INSET policy maker, others maintain that the term "evaluation" 
is itself shorthand by policy-makers (both bureaucratic and elected 
officials) for a spirit of reflectivity towards policy and public 
action. There were other concerns expressed such as the abuse of 
case studies for personal ends and the related problems of how to use 
case study as a tool for professional dialogue and communication between 
various actors in the inservice education and training. Clearly case 
study is gaining an unusual amount of interest as a method for eval- 
uating inservice education and training. Just as clearly, there are 
a number of issues which emanate from a consideration of its use. 

C2 . The role and potential of new quantitative methods to evaluat- 
ing inservice education and training is problematic . Traditional 
quantitative methods based upon experimental designs and matched sam- 
plings (Campbell and Stanley) are little help in evaluating inservice 
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education and . training. But that is not all there is (see Fox; Guba). 
Furthermore, the problem is not necessarily whether one should use a 
qualitative method (e.g. case study) or a quantitative method (e.g. 
survey) to investigate inservice education and training. Instead, 
the challenge is to find appropriate quantitative methods and, in 
particular, a hope (raised in Parlett and Hamilton) that there may be 
some way to integrate quantitative methods and qualitative methods of 
inquiry . 

One considerable aid in resolving the quantitative/qualitative 
debate is the realization that the outcome or consequence of an in- 
service education and training program need not be measured quanti- 
tatively (be it to improve a system, develop a professional educator 
or reach a social aim). Qualitative investigatory procedures such as 
portrayals of impact and in-depth witness accounts of the consequences 
and impinging factors of being in an educational program are recognized 
by many program designers and policy makers as legitimate accounts of 
the intended and unintended outcomes of an INSET program. Just as 
apparent is the possibility that quantitative investigations can provide 
illuminatory evidence of the instructional process used in inservice 
education and training. Time-series is one quantitative method that 
has been used in investigating some inservice education and training 
programs (e.g. Fox; Anglin et. al). Other stochastic procedures (i.e. 
quantitative methods to analyze dynamic change and process) are 
available for use in investigating inservice education and training. 
These include cluster analysis, catastrophe theory and system models 
that invite a dialectical interpretation of behavior. Some of these 
methods, in particular time-series, require concommitant qualitative 
investigations such as case studies to help interpret the results (see 
Glass et. al; Fox et- al). Other stochastic procedures such as cluster 
analysis and catastrophe theory are consistent with the phenomenological 
basis for much qualitative investigation into human behavior and thus 
are capable of being integrated into case study investigations. 

Furthermore, developments and refinements in the quantitative anal- 
ysis of "single-subject research" make it possible to perform appropri- 
ate quantitative analysis on particular cases over time (see Kratoch- 
vill). There are legitimate, highly sophisticated but easily understood 
methods of analysis where the "subject" (e.g. individual, class, school) 
can be analyzed and compared to itself rather than to a control group 
or to matched samples. What is needed is "base-line data, 11 that is, 
a description of the subject before an intervention (such as inservice 
education and training) takes place. 

- Thus, the issues are not whether there are available quantitative 
procedures consistent with phenomenological perspectives of inquiry. 
Instead, the issues are more practical. Can these quantitative proce- 
dures be learned? (The answer in many cases is yes, easily, see Fox 
et. al). What is their relative cost in terms of time, money and 
technical support? (see Hamilton) What do they offer to policy makers 
and program designers of inservice education and training? The answer 
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to the last question is that some quantitative methods can help illu- 
minate our understanding of the process and impact of inservice 
education and training. They cannot, however, be expected to provide 
unassailable or indisputable results. 

C3. The link between the process of inservice education and 
training and its "product" (improved school practice, implemented 
policy or professional development) needs special investigatory atten- 
tion . Traditional process/product studies assume that the linkages 
are known on obvious, logical grounds* Experience is suggesting that 
the relationship between the inservice education and training experi- 
ence and eventual school practice (or implemented policy or profes- 
sional development) is not well understood and needs to be investi- 
gated directly- This is a different problem from those raised by the 
debate on quantitative or qualitative investigations. Instead the 
issue rests upon what would be acceptable evidence chat a particular 
inservice education and training program produced the effects captured* 
The problem in investigating the link between the inservice education 
and training is so great that some evaluators are arguing against any 
"product" or impact assessment. Others, while noting the inconsisten- 
cies between what we thought we knew and what happens, are suggesting 
that more investigatory attention be given a) the supposed linkage 
(e.g. Scriven's "modus-operandi" method), b) an unencumbered view of 
process and product (e.g. Scriven's "goal-free" method), or c) detailed 
investigations of the recipients of the program, the action of the 
program and the events after a program from a variety of differing 
theoretical and experiential perspectives. Process/product approaches, 
then, are being questioned not on quantitative/qualitative grounds but 
rather on grounds that the link between process and product in inservice 
education and training is still not well understood. 

At the same time a process/product approach to evaluating inser- 
vice education and training is very attractive in its bureaucratic 
context. It can help simplify the bureaucratic decision-making 
process and -reduce the political nature of inservice education and 
training. If a process/product linkage is held problematic, then the 
arguments for or against particular policies on inservice education 
and training are even more subject to pressures from professional 
interest groups. Alternatives to process/product designs may require 
more trust between professional groups and more respect for the potential 
of professional dialogue than many policy makers are able to tfiuster. 
Demands for process/product studies of inservice education anC; training, 
however, restrict the choice of an evaluation methodology (both quanti- 
tative and qualitative methods of investigation). Moreover, it avoids 
some of the most difficult questions that must be asked in an evaluation 
of inservice education and training. 

C4* Many of the distinctions we have formed about "formative" 
and "summative" methods of evaluation do not apply to the evaluation 
of inservice education and training . Although these two terms were 
originally formed by Scriven for evaluating curriculum programs, they 



have often beerfused to refer to evaluations of inservice education 
and training as well. Formative evaluations have been referred to 
as diagnostic accounts of the program in operation with "feedback 11 to the 
program designer; summative evaluations place the program into a larger 
context with final reports fed to policy makers outside of the opera- 
tional program. This distinction makes more sense to evaluating curric- 
ulum programs than it does to evaluating inservice education and train- 
ing. An analysis of the program in its larger context is just as crucial 
to the designer of inservice education and training as it is to the 
policy maker. An investigation into the bn-going process of inservice 
education and training is just as crucial to the policy maker as it is 
to the program designer. Thus, the distinction between formative and 
summative evaluation is not helpful if it is used to separate the needs 
of the designer of inservice education and training from the policy ma- 
ker. Both can use descriptions and analyses of the educational process, 
unexpected turn of events and their consequences to the program (i.e. 
formative evaluation). Both can use descriptions and analyses of the 
program and its consequences in a context larger than the program (i.e. 
summative evaluation). Such information is useful to program designers 
and policy makers before, during and after an inservice education and 
training program. 

To summarize the issues emanating from the choice of evaluation 
methodology, there are many ways in which an inservice education and 
training program can be investigated. There are a variety of case 
study traditions. There are quantitative methods that can illuminate 
the process and impact of inservice education and training programs. 
There are quantitative methods of inquiry that can be integrated with case 
studies to form singular, comprehensive investigations of ■ inservice ■ 
education and training. Process/product evaluation and other traditional 
evaluation methods borrowed from curriculum evaluations (such as 
formative and summative) are insensitive to the challenges of evaluating 
inservice education and training. Thus, the essence of this reality 
that a methodology needs to be chosen is that a) there are many approp- 
riate methods available and b) the best of traditional approaches to 
evaluating curriculum reform may not be applicable. This puts an unusual 
amount of responsibility on those who are faced with evaluating an in- 
service education and training program in a particular context. It 
implies that we need to know the available options as well as the 
context in which the inservice education and training program is to be 
conducted. Since these options are not yet a part of our respective 
professional backgrounds, the choice of investigative method is not 
routine. It suggests we may need to educate ourselves about the 
available options, but it also suggests that "experts 11 in curriculum 
program evaluation can not be left alone with evaluating our inservice 
education and training programs. This brings us to the third reality: 
the many meanings of "participation 11 as it is applied to evaluating 
inservice education and training. 
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D. Issues Proceeding From The Many Meanings of Participation 



Collaboration, involvement and who should participate in evalu- 
ating inservice education and training are major concerns of those who 
design the programs or must evaluate their performance and effective- 
ness. If there is one concern that has been consistently raised about 
the evaluation of inservice education and training since the first 
international conference (Philadelphia, July 1976), it is the necessity 
to make an evaluation more than an autocratic enterprise (see MacDonald 
and Walker), Participation in the evaluation of inservice education 
and training is addressed more as a reality that needs to be responded 
to than as an ideal aim to be reached. Just as clear is that issues 
relating to participation in the evaluation of inservice education and 
training are partly resolved in their bureaucratic context and by the 
evaluation methodology chosen. Nevertheless, the issues have bean 
constantly referred to in discussions on the evaluation of inservice 
education and training as major problems that had a life of their own 
independent of the bureaucratic context or of the chosen investigatory 
methods. The following are some of the issues raised about participa- 
tory involvement by those who have performed and analyzed the 
evaluation of inservice education and training in their respective 
countries . 

Dl. The evaluation of inservice education and training programs 
is a special case for educational investigators because all participants 
are adults and all are educators . Those who participate in inservice 
education and training (for whatever purpose) have obvious rights and 
obvious powers as citizens and as organized professionals in OECD membe; 
countries. No participants can be considered as passive recipients 
or treated as powerless clients. In this sense, participation in the 
evaluation of inservice education and training is a political response 
to the reality. To be more specific, it may be political in a bureau- 
cratic sense, where a variety of professional interests need to be 
appeased if a particular inservice education and training program is 
to be put into action. This political sensitivity to the views of 
participating educators is even present when inservice education and 
training is intended to implement social policy. Because of participating 
educators' lack of understanding of the intent or the context of social 
policy, the policy can be significantly altered by their participatory 
involvement. Likewise, issues were raised about how to encourage 
participant involvement when the participating educators were not 
fulfilling their responsibilities (e.g. when improved school practice 
of participating educators was the aim of the inservice education 
and training). Participation was suggested to be a viable strategy 
for encouraging professional reform in those who were considered to 
be "deficient". Other issues were directed at the possibilities for 
participation in large-scale settings. As an example, it was suggested 
by some that it may not be possible (by others that it may be possible) 
to include representatives of participating educators in the evaluation 
of large-scale national progratns in inservice education and training. 
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The concern in- "all these issues is the fact that those participating 
in inservice education and training are adult educators and must be 
acknowledged and respected as such in the evaluation process. It is 
also clear that the purpose, the setting, the bureaucratic context 
and methodology chosen all play a part in determining the range in 
• njeanihg of the phrase "participatory involvement 11 when referring to 
the evaluation of inservice education and training. 

t 

D2. The many roles of educators participating in inservice 
education and training programs make it very difficult to be certain 
about who should be involved . Educators participating in inservice 
education and training may include teachers, administrators, classroom 
paraprofessionals , program evaluators, middle-level agency managers > 
or combinations of these educational roles. This makes questions 
about who should be involved in the evaluation of inservice education 
and training seem more meaningful in a group where different members 
are referring to different programs. (Conflicting answers may only 
indicate that there were different educational roles involved in the 
different programs.) The identification of participants in inservice 
education and training is sometimes more difficult than identifying 
the participating investigators. Inservice education and training 
can be intended to benefit other than the participating educators. 
When the intent is to improve school practice, for example, there may 
be one set of participating educators (e.g. the administrators) but 
the intent is to benefit the work of others (e.g. the teacher and the 
student). In this example, who should participate in the evaluation 
of the program? When the intent is to implement social policy, there 
is even a wider range of those who may evaluate the performance and 
effectiveness of inservice education and training. A list of possible 
participants in the evaluation of these programs is nearly endless. 
Who may be involved in the evaluation of inservice education and 
training programs intended to improve school practice or to implement 
social policy may be more difficult to answer than in those programs 
intended to stimulate professional development. Thus, the many roles 
of educators participating in the inservice education and training 
make it necessary t o address the issue of who should be involved in 
terms of the purpose (s) and setting of programs as well as the 
bureaucratic context and the investigative method chosen. 

33. When participation in the evaluation of inservice education 
and training is interpreted from a bureaucratic perspective, it may 
merely perpetuate already^ existing uneven power relationships . A 
bureaucratic perspective of participation may be "political" in the 
sense of responding to competing professional perspectives. This 
perspective of "political" is particularly unfortunate when inservice 
education and training is intended to improve school practice or to 
implement social policy. Professional interests may receive attention 
to the detriment of others such as citizens and students. Even within 
the profession, the case studies indicate how difficult it is to work 
across already existing power relationships. Teachers working with 
administrators, school personnel with university based trainers, 
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certified evaluators with the unitiated seldom reduce the organiza- 
tionally created distances between the "lower" and the "higher" posi- 
tions. Understanding can occur, respect sometimes, but power is 
seldom redistributed in our evaluations of inservice education and 
training programs. A few have tried and may begin to show the way 
(e.g. in the case studies, Elliott and Olson; in the U.S., Ward and 
Tikunoff; Bussis, Chittenden and Amarel). In Australia, considerable 
attention has been given to redistributing school decision making 
responsibilities through inservice education and training (see Batten). 
For practical reasons, the large-scale evaluation of this enterprise 
did not reflect the same spirit of involvement and perhaps that is why 
the evaluation was so successful. (This did not seem to affect the 
spirit of the inservice education and training.) The suggestion from 
experience in evaluating inservice education and training is that as 
enlightened as we ^bureaucrats may be and as concerned for participatory 
involvement in the evaluation of inservice education and training, 
there exist ..insidious uneven power relationships. More just rearrange- 
ments of power in the assessments of our programs may not be hopeless 
but they certainly are not achieved through rhetoric. 

D ^ • Uneven professional experience and unequal stages of prepara- 
tion for the evaluation of inservice education and training can make 
participation a cursory activity . We may be sincere about encouraging 
participation of the professional underdog or the economically disen- 
franchized la the evaluation of INSET programs. Honesty, however, 
and a candid review of our experiences suggest how serious is the l?ck 
of preparation for the more significant evaluation functions by thos ■■- 
we are trying to involve. Preparation can be accomplished, partici- 
pation can be significant and substantial but it takes time, support., 
attention and patience. Some are suggesting initial training, some 
are suggesting initial periodic support gradually diminishing, others 
are suggesting different levels of participatory involvement depending 
upon previous experience. All suggest a realistic attitude towards 
participatory involvement in the evaluation of inservice education 
and training. Part of this realistic attitude is to make public the 
strengths and weaknesses of the professional evaluator and use them 
accordingly. The eventual success of an evaluation, however, may 
depend on how feasible can the evaluation responsibilities be carried 
out by the various participants. Most successful evaluations of 
inservice education and training programs include a significant support 
system for encouraging participatory involvement. 

In summary, the need to address participatory involvement in the 
evaluation of inservice education and training is real, and seemingly 
crucial issues may be raised around who should be involved. Upon re- 
flection, however, the issue may be receiving more attention that it 
deserves in discussion sessions between educators who may be uncon- 
sciously referring to quite different contexts of inservice education 
and training. For continued discussion on participation in evaluticn 
to be valuable, it may be necessary to have it far more grounded in 
specific contexts. Some issues are general, such as the need to address 




the participatifrg educators as adults and career professionals when 
evaluating the effectiveness of the inservice education and training 
on their own performance. Problems in dealing with traditional, 
organisationally supported, uneven power relationships also is shared 
across many contexts. Other issues are very context bound such as 
the differing interpretations of who are the participating educators 
or who are the beneficiaries . of the inservice education and training. 
Likewise, the discrepancies in evaluation experience between the chosen 
participants in the evaluation may be more or less significant depending 
upon the purpose of the inservice education and training program. For 
greater understanding of the meanings of participatory involvement in 
INSET evaluation, more attention needs to be placed on the contexts in 
which particular evaluations of inservice education and training occur. 

E. Summary: The Embedded Nature of the Mundane Realities 



Issues have been redirected by those who have recently performed 
and reviewed evaluations of inservice education and training in their 
respective countries. These issues have emanated from three different 
mundane realities: the bureaucratic context of evaluating inservice 
education and training, the choice of evaluation methodology and the 
meanings of participation* Each of these realities was experienced by 
the case study writers in their respective countries. Many of the 
problems that case study writers felt were significant and irasc ible 
in their (and their compatriots 1 ) attempts to evaluate inservice educa- 
tion and training proceeded from these three realities. They felt 
that future performance in evaluating inservice education and training 
will depend upon our resolutions of issues like the ones they raised 
from their past experience. Although no issues raised were expected 
to be universally applicable to all evaluations of inservice education 
and training, the three mundane realities were suggested to be shared 
by all of us who evaluate INSET. Furthermore, future performance 
in evaluating inservice education and training can in part be improved 
through an analysis of our bureaucratic contexts, our choices of 
methodology and our meanings of participation. Finally, the discussions 
on the issues have implied that these thr.e realities are related. 

Figure 5 shows ths three realities as three levels, each succes- 
sive level being embedded in the previous one. The primary level is 
the bureaucratic context in which a particular evaluation of inservice 
education and training is conducted. Embedded in the bureaucratic 
context is the choice of methodology used to investigate the program 
and its impact. The meanings of participation applied to evaluating 
a particular inservice education and training program are embedded 
lii the choice of methodology and in the bureaucratic context. Each 
level is a source of many issues. Each reality is capable of being 
addressed on its own as if it were not embedded in another. In fact, 
this is what happened in previous discussions on the evaluation of 
inservice education and training. Issues were not analyzed as emanating 
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Figure 5. The Embedded Nature of Three Mundane Realities in the Evaluation 
of Inservice Education and Training (including three" different 
methods for analyzing issues proceeding from these realities) 
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from a reality, -tfiat was embedded in other realities. The issue of who 
should participate in the evaluations of inservice education and 
training, for example, has too often been raised as if it were not re- 
lated to the bureaucratic context and the choice of methodology. 
Figure 5 suggests that it may be necessary to analyze the issues of 
participation in evaluating inservice education and training as they 
are embedded in other mundane realities as well. 

Figure 5 also shows three ways in which to examine the issues we 
raise from our experience in evaluating inservice education and train- 
ing. First, each issue could be examined separately. The meanings of 
"political" in a bureaucratic context, for example, could be analyzed 
from the perspective of different contexts in different countries. In 
Figure 5 this possibility is shown by the arrows at the bottom of the 
figure. This strategy describes most discussions on the evaluation 
of inservice education and training prior to the seminar of the case 
study writers. Second, the set of issues emanating from a particular 
reality could be examined together. A better understanding of the 
issues and of the reality from which they proceed may result from such 
an analysis. In Figure 5 this possibility is shown by the arrows 
within the levels. Discussions in the seminar of the international 
case study writers is an example of this form of analysis. A third 
form of analysis is to relate a particular issue emanating from one 
reality to issues emanating from other realities. This is shown in 
Figure 5 by the arrov running through the levels. No consideration 
of the challenges in evaluating inservice education and training would 
be complete without this third method of examination being included 
in our deliberations. Although chis has not yet been done, it is 
possible to imagine how such an examination would be performed. 

The intent of Figure 6 is to indicate how issues in one level 
of reality can be related to issues from other levels of reality. 
Figure 6 summarizes the issues raised by the international case study 
writers. Each issue can be examined alone to determine its significance 
to other evaluations of inservice education and training. The issues 
of each column in Figure 6 may also be examined together (along with 
other issues the reader wants to add) to further our understanding 
of the realities we face in evaluating inservice education and training. 
The most important implication in Figure 6, however, is that each issue 
in Figure 6 may be examined as it relates to the issues in the other 
realities in which it is embedded. Thus, if we considered an issue 
from the meanings of participation (e.g. the variety of participants), 
that issue can be analyzed by relating it to issues raised from the 
choice of methodology (e.g. case study method, use of quantitative 
methods, integration of quantitative with qualitative methods) and to 
issues raised from the bureaucratic context (e.g. respect for pseudo- 
science, mundane reports, the bureaucratic meaning of political). If 
we began to perform such an analysis, it is clear that our resolutions 
of issues of participation or of methodology are not particularly 
meaningful until we consider them in terms of our resolutions of the 
issues raised by the bureaucratic contexts of our INSET evaluations. 



54 



ISSUES PROCEEDING 

ISSUES PROCEEDING FROM ISSUES PROCEEDING FROM FROM THE MANY MEAN- 

. THE BUREAUCRATIC CONTEXT THE CHOICE OF METHODOLOGY INGS OF PARTICIPATION 



There may be nothing as 
"political" to a bureaucrat 
as the evaluation of inser- 
vice education and training. 

There is a respectability 
and a legitimacy to pseudo- 
scientific results. 

Information from evaluations of 
inservice education and training 
may be minimally related to 
bureaucratic policy making 
procedures. 

The bureaucratic context may 
determine some of the most 
crucial problems faced in the 
reporting of results. 

We who evaluate have our own 
bureaucratic context. 

A theory-practice dichotomy is 
reinforced, if not created, by 
the bureaucratic context. 

The bureaucratic context is 
' particularly significant when 
evaluating the role of inser- 
vice edu 'ation and training 
in the redistribution or 
maintenance of power. 



Case study, as a focus on 
particular instances, has 
inherent strengths and 
weaknesses for evaluating 
inservice education and 
training programs. 

The role and potential of new 
quantitative methods to evalu- 
ating inservice education and 
training is problematic. 

The link between the process of 
inservice education and train- 
ing and its "product" (improved 
school practice, implemented 
policy or professional develop- 
ment) needs special investiga- 
tory attention. 

Many of the distinctions we have 
formed about "formative" and 
"suimative" methods of evaluation 
do not apply to the evaluation of 
inservice education and training. 



The evaluation of inser- 
vice education and train- 
ing programs is a special 
case for educational inves- 
tigators because all par- 
ticipants are adults and 
all are educators. 

The many roles of educators 
participating in inservice 
education and training pro- 
grams make it very difficult 
to be certain about who 
should be involved, 

When participation in the 
evaluation of inservice ed- 
ucation and training programs 
is interpreted from a bureau- 
cratic perspective, it may 
merely perpetuate aheady 
existing uneven power rela- 
tionships. 

Uneven professional experi- 
ence and unequal stages of 
preparation for the evalu- 
ation of iriservice education 
and training' can make parti- 
cipation a cursory activity. 



Figure 6. Summary of Issues Proceeding From Three Mundane Realities 
in the Evaluation of Inservice Education and Training 
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In short, Figure 6 suggests that there is much yet to be done in the 
analysis of our experience in evaluating inservice education and 
training as we determine how we can do better next time. 

Greater understanding of the realities we face in evaluating 
inservice education and training will need more reflective inquiry, 
more professional dialogue and more cultural challenges to what we 
have supposed was certain. The hope to be found in this report of 
the issues raised by the international case study writers is that 
reflection upon our professional experience is possible, even in 
international settings. Meetings like the one reported here can 
contribute to better performance in the future, however, only to the 
extent that our reflective inquiry is continued and applied. 
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IV. Reflecting in Context: Confronting Wisdom and 
Ignorance in Ourselves and in Our Colleagues 



A. Preparing for Dialogue Between Policy Makers, 
Program Designers- and Evaluators 

"Dialogue is not discussion or debate. It is not a talk show 
or a brain game or a display of expertise — or esoterica — or urbanity. 
It is not a demonstration or an exhange of information. It is, to 
paraphrase Joseph Schwab, a cooperative inquiry directed at (1) the 
formulation of a problem which is not susceptible of convincing em- 
pirical demonstration; (2) the statement of its possible solutions; 
and (3) the methodical consideration of those solutions in terms of 
both common and particular circumstances. It is an instrument — 
possibly the instrument — for continuing investigation of matters 
which lie outside the realm of positive proof. 11 

Mr ton Mayer (1978) 

Deciding how to evaluate inservice education and training will 
seldom be routine or automatic. Whether to evaluate, why to evaluate, 
how to evaluate, by and for whom must be considered carefully. Few 
choices are predetermined by the context of the program; no choices 
are possible to leave to impartial, scientific judgment. Most sig- 
nificant issues that need to be resolved when evaluating a particular 
inservice education and training program will be resolved only through 
personal judgment. In evaluating inservice education and training, 
there are simply no attractive alternatives to dialogue. It is a 
problem, in Milton Mayer's words, that is not susceptible to convincing 
empirical demonstration. 

Consider the three features of dialogue described by Mayer: 1) the 
formulation of the problem, 2) the statement of its possible solutions J 
and 3) the methodological consideration of the solutions in terms of 
common and particular circumstances. The first and second features 
have been referred to in previous sections. Rudiments in forming the 
problem of evaluating inservice education and training are contained 
in the discussions on our reasons for supporting inservice education 
and training. Figure 4, was suggested as one reasonable way 

to form the problem of evaluation according to the three different 
reasons we use to support inservice education and training (to stimulate 
professional development, to improve school practice, to implement social - 
policy). The discuss^n on the mundane realities of INSET evaluation 
and the issues proceeding from these realities is a way to state the 
possible solutions. Figure 6, suggests the range of possible ; 

solutions to the evaluation of inservice education and training* This 
brings us to the third feature of Mayer f s description of dialogue, the 
methodological consideration of solutions in terms of both common and 
particular circumstances. If we are to arrive at solutions to evaluating 
inservice education and training, this is where the most important work 
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is needed. 



Experience is suggesting that our greatest problem in investigating 
inservice education and training is in finding ways to consider the 
solutions. Our greatest challenge is not necessarily in formulating 
the problem nor in stating the possible solution; it is in how dialogue 
between policy makers, program designers and evaluators can be respect- 
ed, encouraged, supported and maintained in such a way that inservice 
educscion and training can be a topic of cooperative inquiry. Forming 
the problem or stating the possible solutions is important but it is 
simply not the only or the major challenge to dialogue. Just as cru- 
cial is the need to find a way in which cooperative inquiry and method- 
ological consideration can take place. If that happens, of course, the 
problem may be reformed and the possible solutions may be restated. 
Thus, I have chosen to make recommendations on how policy makers, pro- 
gram designers and evaluators can better prepare themselves for dialogue. 
Given the attractive possibilities for recommending alternative methods 
for investigation, perhaps I should explain why I consider dialogue 
to be of such significance. 

Previously I had investigated a number of evaluations of a 
national program in inservice education and training (Teacher Corps) 
with the intent to recommend how to do a better evaluation next time. 
In that report I made six recommendations; 

1. Have professionals evaluate the inservice education 
and training program who themselves are responsible 
for determining or carrying out similar programs. 

2. Expend as much effort in making the reports clear, 
interesting and useful as there is expended in 
collecting, analyzing and interpreting the results. 

3. Continually, or at least periodically, include client 
policy makers in on the interpretive process. 

4. Consider the needs of the teacher education profession 
for unique information or unusual perspectives and in- 
terpretation of their practices along with the needs 
of the policy makers. 

5. Choose investigative procedures that can capture 
growth, dynamic interplay, liveliness, and impact 

(as well as the absence of these qualities)* 

6. Conduct evaluations of inservice education and training 
as interpretive efforts rather than research results. 

It is the best advice I have given. A new evaluation of that 
program is now underway, however ? and these recommendations are proving 
to be incapable of being put into action because of two illusions I held. 
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The first illusion was that the most difficult audience to convince 
in suggesting an alternative approach to evaluating inservice educa- 
tion and training were the policy makers. Thus, I wrote the analysis 
and made the recommendations to the bureaucratic managers of the agency 
(Teacher Corps) who would have the authority to choose whether to 
support another evaluation and, if so, po request a new kind of study 
if they felt that was appropriate. I had erroneously assumed that if 
the recommendations were convincing and clear to the policy makers, 
they would be understood and capable of being responded to by the even- 
tual evaluators. This has not been the case. Even the recommendation 
to choose investigative procedures that capture the growth and liveli- 
ness of impact has been understood more by the bureaucrats than the 
evaluators (i # e.it is the bureaucrats who are looking for appropriate 
and challenging investigative procedures). Likewise, the recommendation 
to consider the evaluation as our best interpretations rather than un- 
assailable findings was understood better by some managers of the agency 
than by the evaluators or, for that matter, by the university based 
designers of the inservice education and training. 

As wrong as I was to assume that evaluators could understand and 
respond if agency bureaucrats could, I was even more mistaken in the 
second assumption. As the first recommendations clearly indicate, 
I had assumed that professional dialogue, although it may be difficult, 
was respected. I had the illusion that once policy makers (mainly 
agency heads and middle level managers), program designers and evalua- 
tors got together, professional dialogue on the study, procedures, 
settings, basic questions and the interpretation of results was natural. 
That illusion has proven to be an even greater disaster than the first. . 
It was simply wrong (or, if you prefer, ignorant) of me to assume that 
dialogue on an evaluation of inservice education and training was 
respected by professionals representing differing interests. Suggesting 
that policy makers be involved in the interpretation, for example, was 
assuming that evaluators would respect the interpretations and the 
experience that agency managers were using to make their interpretive 
judgments. Likewise, suggesting that the needs of the teacher education 
profession for illuminative information from evaluation of inservice 
education and training programs was based on similar ignorance about 
how the competing ambitions of program designers and the evaluators make 
dialogue between them extremely cautious. Furthermore, suggesting cre- 
ative reports to policy makers and practitiioners of inservice education 
and training assumed a respect for teacher education that is apparently 
shared by relatively few of us. 

The easiest recommendation to apply in specific circumstances of 
inservice education and training evaluation is the one referring to 
investigative methodology. There are many approaches being developed 
that expect dynamic change to take place, are created to capture change, 
and, in particular, are designed (or can be adapted) to be appropriate 
to the realities we face in evaluating inservice education and train- 
ing. The democratic evaluations of the SAFARI project (MacDonald and 
Walker), the responsive evaluations of Stake, the hypothesis generat- 
ing approach of Elliott, the interactive approach of Patton, the docu- 
menting approach of Perrone, the process-impact approach of Fox et. al, 
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as vail as they'll luminative attitude of Parlett and Hamilton all are 
possible methods for capturing the dynamic interplay of inservice 
education and training. In short, the need is not for new methods 
but for reforming the ways in which we speak to one another across 
professional contexts* Our collective wisdom right now is in method- 
ology, our ignorance is in how to engage in reflective dialogue with 
our colleagues. 

The following recommendations, then are addressed not to the 
choice of investigative method but rather to the preparation needed 
to engage in professional dialogue. Instead of assuming that dialogue 
is natural, I will begin with the understanding that we need to prepare 
ourselves if dialogue across our contexts is to occur. To take our 
professional dialogue on inservice education and training seriously, 
however, there is one underlying assumption that is crucial: What ve 
do as policy makers, program designers and evaluators of inservice 
education and training is a significant portion of what schooling is . 
In short, the assumption is that schooling is what we adults do. 
The reason why this assumption is important to dialogue on the evalua- 
tion of inservice education and training is that, unless we take our 
theories, actions, assumptions and beliefs seriously (that is as if 
they mattered), then all discussion on inservice education and training 
is, by definition, trivial. 

Let me explain this assumption further. Those who ask if teachers 
make a difference are fooling no one. Of course teachers make a dif- 
ference, they are an integral part of what schooling jls^. So, too, do 
principals and headmasters make a difference and (along with other 
school administrators) make schooling what it is in our respective 
countries. But this assumption goes further than that. The analysis 
says that teacher educators (those who design inservice education and 
training) make a difference. The claim is that what is or is not done 
in teacher education may be more influential to what schooling is than 
most of us (teachers or teacher educators or bureaucrats or evaluators) 
are admitting. In addition, the assumption says that what agency heads 
and middle level managers of governmental educational bureaucracies do 
in insexvice education and training is significant to what schools are. 
The ruLe^ and regulations formed by governmental bureaucracies, whether 
they are conformed to or not , are a significant factor in making 
schooling what it .is today. (Meyer suggests that our rules are one 
especially cogent feature of schooling when they are not enforced.) 
Furthermore, the assumption claims that elected representatives who 
make decisions on inservice education and training are significantly 
responsible for what schooling is. It is policies made by our 
elected officials that affect the actions of teacher educators and of 
school personnel that are significant in making schooling what it is. 
Essential to our engaging in dialogue on the evaluation of inservice 
education and training is the view that what we do is what schooling 
is. If we do not consider what we do as being significant then there 
is no reason for dialogue on Qur actions. Instead, we would have to 
continue dialogue only on what others do (for example, on what students 
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and teachers do) rather than on our own actions, what we think, be- 
lieve, know, have experienced or plan to do next time would make 
little difference and would not really be worth much of our attention. 

What I am claiming, then, is that schooling is an adult activity. 
Furthermore, few of us, whether we be policy maker, program designer 
or evaluator-, are very far removed from making the adult aspects-of 
schooling what it is in our respective countries. Teachers are im- 
portant; but so are we. If this seems outrageous to us, I am suggest- 
ing that that is why we have not taken dialogue on the evaluation of 
inservice education and training seriously. We not take ourselves 
seriously. We have little respect for our influence. We do dot think that 
reflection into our own behavior and beliefs really much matters. If 
we thought that what we do as policy makers, program designers and 
evaluators was really important, do you think we would be doing what we 
are doing? Much of what we do is so somber, so devoid of liveliness 
and concern and substance because the only meaning in professional 
conflict is to protect our own turf — the importance of what we do or 
what we think is minimal. Let me give an example. I am going to recom- 
mend that policy makers reflect about their own views of social science 
and, if it is only a Newtonian, positivistic view, then they should 
consider some alternative views of science. The only way that a 
recommendation to policy makers 1 views of science makes any sense at 
all is to assume that it really matters what we as policy makers do 
and what we as policy makers hold as theories, beliefs or amorphous 
ideologies . 

If we assume that what we do and think is significant to what 
schools are, then it is natural to place some importance to investigating 
ourselves. Whether we be policy maker, program designer or evaluator 
the understanding we bring to evaluating -inservice education and training 
includes our theories about instruction, our theories about school change, 
our theories about professional development, our theories about the role 
of education in our respective social and political contexts, our theories 
of social reform, our theories about evaluation. These theories are 
worthy of our reflective inquiry to the extent that what we do matters. 
Furthermore, our ignorance and misunderstandings in these areas are 
significant only to the extent that what we do matters. As we disregard 
our own work and our own progress as being worthy of serious inquiry, 
then we disregard not only what we know but what we do not know as well. 
We become embarrassed by our ignorance instead of interested in it. 

The following recommendations assume that there is reason to be 
interested in both the wisdom and the accompanying ignorance we hold 
as policy makers, designers and evaluators of inservice education and 
training. The reason is that the wisdom and ignorance that makes up 
our brands of professional work matters to what schooling is. To 
the extent that this assumption is natural and easy to take, these 
recommendations may appear superfluous. They are not, for example, 
meant to be given to teachers or even to local school administrators 
such as principals and headmasters. For them, there is already a 
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respect for their influence on schooling and thus, there exists a 
variety of sophisticated methods for encouraging reflection on their 
theories and actions and for supporting their professional dialogue. 
The recommendations are to the rest of us, we policy makers, program 
designers and evaluators of inservice education and training who are 
removed more physically from classroom practice and to whom this as- 
sumption has seldom been applied. 

In short, these recommendations are given for those of us parti- 
cipating in "backroom decision-making, " where decisions about inservice 
education and training and its evaluation are made away from the class- 
room. My experience suggests that this not only happens, but can happen 
for good reason. My experience also suggests that it is in these cases 
that reflectivity and professional dialogue are very difficult, signif- 
icant to the performance of the evaluation and can use some suggestions 
for engaging in cooperative inquiry. The following recommendations, 
then, are made to policy makers, program designers and evaluators in 
order that they can be better prepared to confront themselves as well 
as one another as they undertake the evaluation of inservice education 
and training. The recommendations refer to ways in which the inservice 
education and training program and its evaluation can be placed into 
context by the policy makers, program designers and evaluators. Just 
as important for preparing for dialogue and the consideration of possi- 
ble solutions, are recommendations given to the three parties for 
reflecting upon their own wisdom and ignorance that is significant to 
the evaluation of the inservice education and training and for confront- 
ing the wisdom and ignorance that others may bring to the same enter- 
prise, _ ------- - - - - - ~~ 



B. Recommendations for Preparation by Policy Makers 



A policy maker of inservice education and training is responsible 
for identifying why inservice education and training is to take place 
and whether an evaluation study is appropriate. Thus, it is up to the 
bureaucrat or elected official to articulate why an evaluation is or 
is not being conducted. If a policy maker chooses not to make that 
decision, (s)he determines who will (e.g., the designers of the inser- 
vice education and training ). Most important, a policy maker must 
articulate how an investigation is to relate to policy. In short, a 
policy maker is responsible for placing the problem of evaluating in- 
s ? rvi 5 e education and training into context in any continued dialogue 
with program designers and evaluators. This includes making it clear 
to all participants in an INSET evaluation why inservice education and 
training is taking place, why there may be an evaluation and what makes 
the evaluation important to the aims of inservice education and to other 
social or educational policies. While placing the opportunities for the 
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evaluation into a realistic context, the policy maker may be particu- 
larly responsible for encouraging risks to be taken in the evaluation 
enterprise. 

The following are recommendations for policy makers who expect to 
have dialogue with program designers ap s d evaluators about the evalua- 
tion of particular inservice education and training programs. The pur- 
pose of these recommendations is to make it more possible for a 
policy maker to participate in a focused and illuminative consideration 
of the evaluation of inservice education and training. In preparing 
for dialogue with program designers and evaluators, it is recommended 
that policy makers a) place the evaluation of the inservice education 
and training into context, b) reflect upon their own understanding of 
evaluation and, in particular, of how evaluation may or may not be 
used in this context and c) how to confront the understanding, the 
wisdom and the ignorance of those who may be expected to evaluate 
inservice education and training. The eventual success or failure 
of an evaluation of inservice education and training may depend on 
the extent to which a policy maker has prepared for dialogue with the 
program designer and evaluator. 

The following recommendations are suggested to 'policy makers for 
placing the evaluation of the inservice education and training in its 
context : 



Bl. Articulate the reason you support inservice education and 

training (to stimulate professional development, to improve 
school practice, to implement social policy) . Determine if 
there are diffeier± purposes and aims of the programs. Use 
Figure 4, to place your view of the particular inservice 

education and training program in comparison with other 
possible programs. If possible, describe the probable set- 
ting (e.g. single school, multiple schools, ad hoc). In 
short, be as specific and articulate as possible about the 
context of the inservice education and training as you see 
it. 

B2. Consider the efficacy of the evaluation to the inservice 

education and training context . If the intent of the inser- 
vice education and training is to improve school practice, 
remember there are many examples of previous successes in eval- 
uation. If the intent is to stimulate professional development, 
experience suggests that evaluation should not b e an indepen- 
dent function. Instead of funding for evaluations, it may 
be better to fund the creation of better visions of what pro- 
fessional development may mean and on creating ways of reaching 
these visions. If the intent is to implement social policy, 
the evaluation will be costly. Costly because so much needs 
to be done and so many creative energies need to be expended 
to make an evaluation of inservice education and training 
intending to implement social policy useful and insightful. 
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There are few models of success in evaluating inservice 
education and training programs intended to implement 
social policy but it is here where some of the most 
crucial questions for a policy maker are answered. 

B3. Con sider the issues emanating from the realities of an 
£. valuation of inservice education and training . Refer 
to the issues raised in Figure 6. - Analyze your 

views of the issues. Determine which issues you think 
are top priority, which need to be resolved and how they 
may be related to one another. , Determine which issues may 
be most risky, given the context of the inservice education 
and training and your understanding of the bureaucratic 
context of its contemplated evaluation. Decide which risks 
you are most willing to undertake (by considering their 
priority). Try not to make quick generalizations on issues 
emanating from the choice of methodology or participatory 
involvement:. For example, do not expect all quantifiable 
analyses to be convincing nor all qualitative descriptions 
to be sympathetic to the participating educators. Do not 
feel you have to rely on quantitative analysis for analyzing 
outcome and qualitative analysis for analyzing process. 
Leave those issues open for later dialogue. 

B4. Determine- where your responsibilities and continued involve- 
ment may be the most useful in the analytic process of the 
evaluation and in the resolution of issues emanating from 
the realities of the evaluation . For example, when the intent 
is to implement social policy, your involvement may be crucial 
when the intent is to stimulate professional development, 
it may not be so crucial. In resolving issues emanating from 
the bureaucratic context, your views may be essential. In 
resolving issues emanating from the choice of methodology, 
your views are relevant but open for challenge and debate. 

The following recommendations are made to policy makers for re- 
flecting about their own understanding, experience and ignorance about 
inservice education and training: 

B5 . Ask yourself what would happen if you suspended your belief 
in science and increased your belief in a process whereby 
information is negotiated . Determine to what extent you 
could accept and would use an investigator Is best ' judgments 
rather than absolutely convincing empirical evidence. 
Identify your own assumptions about evaluation. Decide what 
assumptions and views you may allow to be questioned, what 
may not be questioned. Be able to articulate your answers 
in some detail. 

B6. Construct your own view of policy and the relationship of 

information to policy . Recreate, as best you can, the policies 
made that relate to your consideration of inservice education 
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and training. Determine what policies are still open for 
consideration, what the options may be and what a reform- 
ulated policy might look like. Describe the process used 
to change policy and the nature of the evidence that may 
be acceptable in that change process. Be candid and very 1 
clear about the timing necessary for that evidence and its 
interpretations. 

B7 * List the alternatives you see to inservice education and 

training. Determine how important the inservice education 
and training is in terms of its purposes and in terms of 
the alternatives. List the alternatives you see to evalu- 
ating inservice education and training but still gaining 
the kind of information you may need. Using the above as 
a basis for consideration, analyze the funds available for 
reaching the purpose, for performing the inservice education 
and training and for evaluating the inservice education and 
training. Never make decisions on funding INSET evaluation 
on any rule of thumb (e.g. 10%). Always judge the signifi- 
cance of an evaluation upon its usefulness to the aims of 
the inservice education and training. 

58 • Ask yourself if you can doubt your own rhetoric for choosing 
inservice education and training . If not, do not evaluate. 

The following recommendations are sug&e^ed to policy makers for 
confronting the evaluators and determining whether an evaluation can 
be done properly: 

B9. Be very selective in who is chosen to evaluate inservice 
education and training . Remember, you are asking for an 
evaluation of a program where adults are the major recipients. 
Evaluators must be able to work on an adult to adult level. 

B1 °* Look at what the evaluators have done in the past . See if 
their reports and reporting procedures make much sense to 
you. If they are difficult to . understand, you will want 
someone else. 

B11 * Do not accept all the solemn pronouncements of evaluators 
towards their methodologies . Methodology is important but 
it is. not always as important as evaluators contend. Figure 
5, \ > j?afe^ s $3 s K v suggests that more important than methodology is the 
bureaucratic context in which the methodology is embedded 
(both the bureaucracy of the INSET and the bureaucracy of 
the evaluation team). Be prepared to discuss methodology 
in terms of the inservice education and training context 
(purpose and setting) and the evaluation issues. In short, 
be prepared to doubt the rhetoric of the evaluator just as 
you should be prepared to doubt your own rhetoric. If you 
are intimidated and, cannot confront the evaluators on their 
own level, the evaluation should not be performed. 
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B12^ If social policy \s to be implemented through inservice edu- 
cation and training, make policy an obvious part of th e for- 
mal request for an evaluation design . Do not ever leave"an~ 
evaluator alone to design, implement and report an evaluation 
relevant to social policy without your periodic involvement: . 
It is necessary that you be involved and crucial if you are 
co gain information relevant to social policy. Require evi- 
dence that the evaluator can work in > policy relevant situ- 
ation and, in particular, can produce in a bureaucratic 
concext . 

B13 - ..Jj^i,giIF£.J.jlT ± others, bo th INSET d esigners a nd evaluator s, 
are villii.g to address some of the harder qu^ c tions that you 
would like asked . The more risks you are willing to rake in 
the INSET evaluation,, the more crucial your choice of evalua- 
tor. The creative gathering of data, the creative us?, of 
analysis and interpretations and a creative approach to re- 
porting in large part depend upon previous experience and 
the power of the investigator, not the .over of the design. 
Get to know you investigator* Ask for sa pies of the person's 
previous work. Ask for evidence that the risks you are willing 
to take are also risks that the investigator is willing to 
take, has taken or can be expected to take with a fair 
degree of success. 

A policy maker's role in ar ' continued dialogue on the evaluation 
of inservice education and training will depend ci how well they have 
analyzed their own views and can articulate them to the program designers 
and the evaluators. It has been suggested that this articulation can 
be helped by referring to th^ intent of the inservice education and 
training and how the INSET purposes relate to other policy. It has also 
been suggested that' identifying evaluation issues according to their 
priority and to the degree of risk that is expected will be helpful to 
the articulation of a policy m.iker 1 s intent . In addition, healthy skep- 
ticism towards the claims of any evaluator may be necessary in the con- 
sideration of what can be done. The more the policy maker can challenge 
evaluators on methodological and participatory issues- the more that 
evaluation may eventually be useful. Professional dialogue and mut.ial 
understand!:^ on the evaluation of inservice education and training 
will depend upon the ability of policy makers t confront evaluators n 
issues that have previously been considered to be the evaluators 1 profes-*- 
sioral domain. Likewise, it will depend upon the possibilities that 
enough has been shared by the policy maker for his or her owi: 'omains 
to be understood and critically analyzed by the progran designers and 
evaluators. •' 



C. Recommendations for Preparation by Program Designers 

A designer of an inservice education and training pro "am is responsi- 
ble for seeing that the instructional intentions of a program are reached. 
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What is important to the teacher educator or staff trainer is that the 
evaluation be as applicable and useful to the instructional intentions 
as is possible. To do this, the design-. of the program needs to assess 
the relationship between the instructional intent and the ultimate aim 

of n^! erVi r/ dU ? aCi ° n Crainln 8- ^n, for example, stimulation 
of professional development is the aim,. the program designer may be in- 

;i e L nV °i ^ idenCif y in S Che hard questions to be asked in the 
evaluation. When the intent is to implement social policy, however 
the program designer's instructional purposes are necessary to consider 
but are less crucial to the more fundamental questions to be asked in 
the evaluation (such as who is benefitting?). 

In preparing for dialogue with the policy makers of inservice 
education and training (usually middle level agency managers) and the 
evaluators, it is important that the designers can articulate their 

ornv^r J"* 11 ' 10115 3nd Can idencif y where an evaluation study may 
provide insight to those intentions. This implies that INSET designers 
must be able to articulate their theories of professional development 
and their operating assumptions being used to design the instructional 

*»service education and training. Thus, just as was 
suggested for policy makers, the program designers need to understand 
tneir own positions, intentions and context well enough to articulate 
them clearly to policy makers and evaluators. They must also be able 
to understand the unique character, demands and possibilities of eval- 
uating inservice education and training well enough to relate these 
possibilities to the educational intentions. 

The following recommendations are made to program designers preparine 
to answer questions about the role of evaluation in the inservice educa- 
tion and training design. As was the case in the recommendations made to 
the policy makers, it is recommended that program designers a) place the 
evaluation of the inservice education and training into context, b) re- 
ject upon their own understanding of evaluation and, in particular, 
of how evaluation may or may not be used in this context and c) consider 
now they will confront the understandings of policy makers and evaluators 
when the investigation of the inservice education and training is being 
determined. The extent to which an evaluation of inservice education 
and training will, be useful to the instructional intent will depend on 
the extent to which the program designer prepares for dialogue with 
policy maker and evaluator. 

The following recommendations are suggested to program designers 
for placing the evaluation of inservice education and training in its 
context: 



CI. 



Identify the inten t of the inservice education and training 
(to sti mulate professional development, to improve school 
practice, to impleme nt social policy) . Determine whether you 
see different purposes and aims of the program. Using Figure 
4 » indicate where you think the inservice education 

and training program is located. Consider the implications 
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of tfiis location (refer to Figure 3 ) <> Make sure 

your instructional intentions and the theoretical context 
of your instructional design are considered and can be 
articulated to others. Describe what you think is the 
optimum setting for inservice education and training, given 
the context that you have just described. 

Consider the efficacy of evaluation not only to the instruc- 
tional design but to other issues suggested by the INSET 
context . If the instructional intuitions are to stimulate 
professional development, for example, and the aim of the 
inservice education and training is for, that purpose, then 
evaluation can be singularly focused. If, however, the 
instructional intentions are towards one end (e.g. to improve 
school practice) and the ultimate -.im ic toward another (e.g. 
to implement a social policy), then there is a question of 
evaluation focus. Should it be on the instructional design 
or the social policy? It is up to the designer, then, to 
identify clearly the instructional priorities for evaluative 
information and to be practical about the resources necessary 
to provide this information. The designer must help create 
an understanding of the feasibility of an evaluation to be 
both policy relevant and professionally illuminative. This 
is particularly important when the instructional intent of 
the inservice education and training is towards one purpose 
and che ultimate aim goes beyond that. The program designer 
should entertain alternatives for gaining information needed 
for implement in g" that design other than through an evaluation 
study. 

Consider each of the three levels of evaluation reality: its 
bureaucratic context, its methodology and participatory ^in- 
volvement . Analyze your views of the issues emanating from 
these three? levels. Perhaps you will need to give special 
attention to participatory involvement since those issues will 
relate directly to the instructional intentions and the theo- 
retical context of the INSET design* Ea prepared to discuss 
issues emanating from the bureaucratic context of evaluation 
ant the methodological considerations as well. Those issues, 
too, may be related by you to your priority areas for investi- 
gation. For example, to what extent can you see quantitative, 
qualitative or an integration of different analytic techniques 
relating to your theories of inservice education and training 
instruction or your plans for implementing these theories? 

Determine what your involvement in the evaluation ma y be . 
Consider your needs for investigative inquiry and t..e possibil- 
ities that your in'volvament in the evsluation can be useful. 
For example, your involvement in the evaluation of a program 
whose aim is professioral development may be particularly sig- 
nificant. Your involvement in an inservice education and 
training evaluation intended to implement social policy may not 
be as significantly fel: by you or by the policy makers. 
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The following recommendations are made to the program designers 
for reflecting about their own wisdom, experience and ignorance in 
evaluating inservice education and training: 



C5. 



C6. 



C7. 



C8. 



State your assumptions ab out the instructional aspects of 
inservice education and training . Be articulate about the 
theory you hold about professional development and how 
inservice education and training relates to that theory. 
^ Identify, if appropriate, your theories of school change 
and your theories of social reform. In particular, state 
how inservice education and training may be related to 
either aim. Be candid. If you hold competing theories 
of professional development, let that be known. If you 
are not comfortable with your theories of school improvement 
or of social reform, make that clear. Your placement of 
inservice education tradition may be necessary to continued 
dialogue. Although policy makers and evaluators will have 
their own assumptions about inservice education and training, 
it is your assimptions that will most likely be the focus of 
the dialogue. 

If profes sional development is an intent of the inservice 
education and tr aining, state your visions of what profession- 
al develo pment may be . There are many possible visions of what 
a teacher can be developing towards (e.g. an action-researcher, 
a developer of instructional theo-ies, an implementer of 

researctl1 arid development ^ conclusions).. _ Be articulate about - 

which visions you hold and how the instructional intentions 
of the inservice education and training are related to those 
visions. Be candid about the ambiguous links between the 
instructional intentions and the vision(s) for professional 
development. Be creative in constructing your vision and 
linking the inservice education and training to the vision. 
Be practical about the resources necessary to reach the vis^n. 
Resources may be more than instructional. Necessary resources 
may include policy such as professional time. 

State the educational theories that influence your inservice 
education and training design . Place the design and its pro- 
cedural components into an educational theoretical context. 
If you see alternative theories, competing titeories or 
alternative implications for practice, state these clearly. 
The evaluation may be most illuminative to the theoretical 
basis for the educational design of the inservice education 
and training. Give examples of lessons you have learned in 
the past and how these have helped you refine your theoretical 
perspective. 

State your own theories of how educational evaluation relates 
to instr uctional design . What evidence is most persuasive to 
you? What investigative processes do you see best relating 
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to your educational design and instructional intentions? 
What investigative procedures do you see stretching your 
own understanding and perspectives of inservice education 
and training? What do you see as the most convincing kinds 
of evidence for influencing the instructional design? Is 
it other participants 1 points of view, is it the recipients' 
performance, is it the link between the aims of the inser- 
vice education and training? Answers to these questions 
will specify what you see as important features of an 
evaluation study jLf it were to aid you in designing inser- 
vice education and training. 

C9. Be articulate and clear about what you may need to know to 
design inservice education and training better . Is it to 
have more clearly articulated theories? Is it to consider 
more exciting ways of knowing? Is it to have more precise, 
detailed and candid descriptions of the actual instructional 
process? Is it to have more views of what the developmental 
process may be? Your answers to these questions will have 
significant implications to what you want from an evaluation 
study. Moreover, it would help to state how you think you 
can best be stretched in those areas where you have the most 
concern. For example, would you be stretched by an unusual 
reporting style or by the data itself? Would you be stretched 
by a creative interpretation of the data or a significant 
review of the literature? Eventual usefulness of an evalua- 
tion study to the construction of your inservice' education 
and training program will depend" upon your abilities to 
articulate these concerns. 



The following recommendations are suggested to program designers 
for confronting the policy makers and evaluators and determining if the 
evaluation study may or may not be helpful to your instructional inten- 
tions : 

CIO . Be prepared to address other than your instructional intentions « 
Do not assume that the major purpose of an inservice education 
and training evaluation is to help the instructional design. 
There may be other priorities (e.g. policy issues) for inves- 
tigation. Make sure you can talk to policy issues and to 
specific methodological issues as well as to the instructional 
design. 

Cll. Relate your priorities for information to those of the INSET 
policy maker . Assess the policy maker's needs for evaluative 
information. Assess also the policy maker's openness to and 
understanding of ypur priorities. For example, if you deter- 
mine it is your theories of instruction that need investigative 
attention, to what extent might this need be shared or under- 
stood by the policy maker? To what extent can you help in 
relating instructional needs to other policy needs? What 
instructional theories are held by the policy makers? 
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C12. ysess th e potential evaluators . What kind of work have they 
done in the past? Is there a possibility for their addressing 
your priorities (e.g. theoretical basis for INSET design). 
What chances do you see that the procedures used in the Lst 

ZJttiT °/ S te US6ful C0 y0U (e -§- data gathering, 
analytic procedures interpretation, reporting)? wW theories 
of inservice education and training are held by the evaluators? 
To what extent can you influence the assumptions and viewpoints 
of the evaluators so that they can better understand and relate 

"Rations? Determine what you Think is a realistic 
relationship between the evaluators and the program you de- 
signed. Should they ideally be part of the instructional 
program or can they more usefully perform their investigation 
as outsiders? To what extent do you see the evaluators being 
like asked?" 55 ^ ° f ^ < * uestions that y°" would 

inten^o e ns US wi f n n H" S °J " eValuation to designer's instructional 
his o r ^r itltnr * T\ abllities ° f «*e designer to articulate 
KnLw ^ intent ^ns and the theoretical basis from which they come. 
Knowing the relation you draw between professional development and the 
till TT ° f SCh ° 01 P ractice ' for —Pie, may improve the coneidera- 
au^nf 5° ] nvesti Z ate - Kn°wing the educational theories that are 

guiding your design or the conflict and ambiguities that you feel in 
applying these theories is essential if an investigation is to provide 
you insight. The usefulness of an evaluation for the INSET designer 
-JJ S^ .»-.fe.p 0t ential^o.address the instructional vision 
of the design This is a particular challenge to the designer, especi- 
ally when professional development is the purpose of the inservice educa- 
tion and training The visions now held for professional development 
are seldom articulated, making our creation of better visions, our 
lessons learned from experience or our ability to share across experi- 
ence that much more difficult. In short, the role of the program 
designer to the evaluation of inservice education and training is very 
similar to that of the teacher to classroom research. It is up to the 
practitioner, the designer of the inservice education and training, to 
help policy makers and investigators understand the theoretical basis 
tor the inservice education and training practice. Professional dia- 
logue with policy makers and evaluators will also depend upon the ability 
of designers to confront the policy and methodology from an educational 
perspective. This will include tm. necessity for the designer to open 
up his or her domain of instructional practice for critical analysis 
by policy makers and evaluators. Dialogue will depend upon the possi- 
Dixity that other than the program designers Can address critical 
educational issue related to the instructional design of the inservice 
education and training. 

D. Recommendations for Preparation by Evaluators 



An evaluator is responsible for identifying possible evaluation 
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strategies and ""their relative strengths and weaknesses when considering 
the evaluation of a particular inservice education and training program. 
An understanding of the underlying assumptions of a variety of evaluation 
alternatives will usually have to cqjne from the evaluator. Likewise, 
a realistic attitude towards what may ibe feasible and what may not be 
feasible in the evaluation of a particular inservice education and 
training program is the responsibility of the evaluator. This means 
that an INSET evaluator must understand many evaluation methodologies 
and not just be one kind of evaluator as was once implied by Stake 
(OECD, 1976). Understanding a methodology includes being able to articu- 
late the underlying assumptions, operating procedures, necessary re- 
sources and viable contexts in which it can be conducted. It will be 
the evaluator' s responsibility to see that the efforts made to investi- 
gate an inservice education and training program are well spent; that 
the risks identified by the policy makers and theoretical, ambiguities 
and contradictions noted by the program designers can be addressed. 
Not only is the evaluator responsible for having a range of craft options 
available to use in a particular context, it is also necessary that these 
options be understood, be confronted and be considered by the policy 
makers and designers of inservice education and training. 

A successful dialogue on INSET evaluation will depend in very 
large part on the ability of the evaluator to speak in plain language 
about a variety of evaluation methodologies. The INSET evaluator must 
ensure that methodology can be a natural common ground for discussion 
where the theoretical bases, conceptual intents and requirements, for 

practical resources of a variety of methods become part of the shared 

understanding of all participants in the dialogue. In short, the mys- 
tique of the evaluator must be broken in order that more significant 
issues than methodology can be considered in dialogue with policy 
makers and program designers. 

The following recommendations are made to evaluators to help them 
prepare for dialogue with policy makers and program designers on the 
evaluation of inservice education and training. As was the case in 
the recommendations made to policy makers and program designers, it 
is recommended that evaluators a) place the evaluation of the inservice 
education and training into context, b) reflect upon their understand- 
ing of evaluation and, in particular, how the evaluation of this pro- 
gram is related to their past experiences and professional ambitions 
and c) to consider how they will confront their own lack of understand- 
ing as they confront the understandings of policy makers and program 
designers when determining how the investigation will be conducted. 
The extent to which the exciting state of the art in evaluation method- 
ology is considered and used appropriately in the evaluation of par- 
ticular inservice education and training programs will depend upon the 
extent to which the evaluator is prepared for dialogue with the policy 
maker and program designer. 

The following recommendations are suggested to evaluators for 
placing the evaluation of inservice education and training in its context: 
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D2. 



D1 - Make clea r your interpretation of the reasons for the inser- 
vice educa tion and training program, specify which you con- 
sider prior ities for investigation . Placing the program in 
Figure 4, state which intentions you see as being 

most important for investigation. State whether you see some 
as purposes, some as aims of the program. For example, if 
you see stimulating professional development as a reason for 
the inservice education and training, is that the ultimate 
aim of the program? Or if implementing social policy is the 
a;;m of the program, are the improvement of school practice 
or the stimulation of professional development secondary 
purposes? Consider how you see an evaluation study addressing 
theintentions of the inservice education and training. Which 
purpose do you think it should emphasize? Why? How will it 
investigate that purpose? Do you see looking at the purposes 
and the aims of the program? How will this be done? What 
else besides the inservice education and training will be the 
focus of your inquiry? 

Place the inservice education and training into its probable 
setting ( single school, multiple school, ad-hoc) . Given the 
setting and the intentions, what do you think is the best that 
can be done to evaluate the inservice education and training? 
Have you worked in this setting before? If so, state what 
you learned works best and why. Look into the possibilities 
for doing something more useful than you have done before. 
Given the setting and the intentions , do you" even think that 
an evaluation study could be useful? Determine how you think 
you should relate to the various persons within that inservice 
education and training setting; assess to what extent you can. 

D3 - Analyze your approaches to the three mundane realities to 

evaluating inservice education and training (its bureaucratic 
context, choice of methodology and participation ) . Consider 
in detail the methodologies available, remembering you are 
the person most responsible for having policy makers and pro- 
gram designers understand the range of options. Analyze the 
methodological possibilities in terms of the bureaucratic 
context of the evaluation and the participatory realities. 
Be fair about the bureaucratic context — both yours and others'. 
Be creative in determining what you think is possible given 
the realities and the issues emanating from these three 
sources (see Figure 6)0 Determine to what extent 

you can be flexible in readjusting the method you think is 
most desirable. 

D4 * Consider how the evaluation can be policy relevant^ illumina- 
tive and participatory. There are, of course, many examples 
of illuminative studies and the conditions under which they 
are most feasible. There are also an increasing number of 
very good examples of investigations with various interpretations 



of participatory involvement . You can judge th« possibilities 
for success in being illuminative and participatory upou some 
previous experience in addition to your own. Policy relevance 
is another problem. You may not have many examples of success 
here. Here, perhaps, you must be a pioneer, analyzing the 
policy context of the inservice education and training and 
determining how, this time, the investigation may relate to 
imminent policy decisions or be used in future considerations 
by policy makers. 

D5. Place your view of the evaluation of this inservice education 
and training program into your professional career . To 
what extent do you see possibilities in the evaluation study 
for improving your own professional understanding of 
evaluation, of INCET or of public policy making? If the study 
does not further your own career, to what extent do you think 
you will really be involved? The complex demands of performing 
an INSET evaluation may make your involvement not worth the 
necessary efforts unless you can gain more than a temporary 
livelihood from the enterprise. 

The following recommendations are made to evaluators for reflecting 
about their own positions on the evaluation of inservice education and 
training: 

D6. State what you have done before in educational evaluation . 

Be candid and clear about the study designs, data collection 
procedures, analyses and reporting procedures you have 
actually used. Place previous activities into the realities 
of evaluating inservice education and training as much as 
possible (e.g. into its bureaucratic context and meanings of 
participatory involvement as well as choice of methodology). 
Identify both the best things and the worst things you have 
done. List some of the lessons you think you have learned 
from your past evaluation efforts* Relate these to the 
realities and issues shown in Figure 6. 

D7 . Place what you have done before into a wider context. In 

methodology, for example, indicate the range of possibilities, 
both quantitative and qualitative. For example, what tradi- 
tions of case study have you used (if any): naturalistic, 
theoretical or historical? What quantitative analyses do 
you see available and which ones have you used? What is the 
the nature of the study designs you see available and which 
ones have you used? Here perhaps a grid (e.g. House, 1977) 
may help the articulation of possible evaluation apprca-hes. 
So, too, may the political categories of KacDonald (oureau- 
cratic, autocratic and democratic). Of those methods , designs, 
approaches to evaluation that you have not tried, indicate 
which you think you understand (or are willing to understand) 
well enough to try in the future. Indicate which ones you 
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are not willing to try. If there are some possibilities 
that you have not tried in the past but are particularly 
interesting to you now, state what you think you would need 
to do (if anything) to use that procedure. What other 
accounts have you read? What other reports have you care- 
fully reviewed? What other procedures have you observed in 
practice other than your own? 

D8. State your own assumptions about inservice education and 
training . State your own assumptions (or observations) 
about how inservice education and training policy is made 
and how evaluative information is or is not used. State 
your theories of the educational process, particularly 
your theories of professional development. State your theories 
about how schools improve, about how social reform is or is' not 
accomplished through social policy. Show how your views of 
evaluation relate to your assumptions about INSET, about 
professional development, school improvement, social reform and 
about what knowledge is and how it is generated. Don't expect 
to be complete or consistent but gain a better idea of where 
you feel most comfortable about your understanding and where 
you do not. Identify ways in which you can build your under- 
standing about INSET, professional development , school improve- 
ment and social reform. Consider holding a theory of ways 
of knowing as a responsibility of your craft. 

D9. Review your own view ..of science and., its relationship, to the 

kind of investigations you can probably^ perform in evaluating 
an INSET program . Examine, for example, what evidence you 
anticipate gathering in an INSET study and the nature of the 
conclusions, implications and recommendations you may reach. 
To what extent can they be related? Consider the Toulmin 
view of science, science as dialectical inquiry rather than 
empirical positivism. Consider the Feyerabend view of science, 
science as the creation of outrageous statements rather than 
a deliberate attempt to reduce the number of false theories. 
Be playful, not so self-indulgent about your views of evaluation 
as science. Be less somber about the responsibilities you think 
you hold as a generator of knowledge about INSET. Be honest 
about your failures as a scientist but equally as honest about 
your evaluative abilities as a persuador (see House; Perelman 
and Olbrects). 

D10. Identify your own bureaucratic context and what it may mean 

to what you can do and how you can approach an INSET evaluation . 

Dll. Try to imagine the most simple and creative ways to evaluate 
inservice education and training . Don't imagine yourself as 
a major figure in evaluation theory nor a developer of major 
innovation methodologies for the profession. You are not a 
pioneer, but a scout who can tell the rest of the party what 




is otftr there (see Roland Barthes). The challenge is to 
consider in dialogue how an inservice education and train- 
ing program can be evaluated, using the best of what is 
available. The invention of new methods is seldom neces- 
sary. 

The following recommendations are suggested to evaluators for 
confronting the policy makers and program designers on the viability 
and usefulness of an evaluation to their intentions for the inservice 
education and training program: 

D12 Know the policies in which the inservice education and train- 
ing is embedded . Analyze the extent to which you understand 
and can address the policies in an evaluation study. Analyze 
the extent to which the realities of the bureaucratic context 
and the setting of the inservice education and training pro- 
gram may restrict the possibilities for policy-relevant con- 
clusions and recommendations. Decide if it is possible for 
you to be policy relevant within this context. If you have 
doubts, share them with the policy makers. They may have 
other ideas of what would make the evaluation relevant to them. 



D13. Prepare to understand the policy makers and instructional 

designers . Learn their views of evaluation, their theories 
of professional development, their interpretations of science, 
investigation and knowledge. Gain an understanding of the 
contexts and pasts of the policy makers and designers that is 
necessary for you to address them at their own level's of con- 
cerns, experience and understanding. Realize, of course, that 
they may have more experience in evaluations than you and that, 
from their perspective, many of the past evaluations have been 
failures. Determine to what extent they may be flexible in 
their consideration of how events can be understood, described 
and their meanings communicated. Consider their views of in- 
struction and how evaluation is or is not contained in their 
views. Make sure you understand the designers 1 theories of 
professional development and can place (revise, recreate) your 
views of evaluation into that perspective. For example, you 
may want to place your methods for capturing the intentions, 
processes and consequences of the inservice education and 
training into the designers 1 theories of professional devel- 
opment. In short, you must be prepared to understand the 
realities and perspectives of policy makers and designers on 
their own terms. You must be able to articulate the options 
and possibilities you see in the evaluation of inservice edu- 
cation and training into their terms. As sponsors of the 
study, they must gain insights that they are able to recognize 
in their work. Never underestimate the capabilities of the 
policy makers and the program designers to understand the issues 
in evaluating inservice education and training. 
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D1 ^* Determine the extent to which the significance of the poli- 
cies and the challenge of the instructional designjmay stretch 
your own capabilities in evaluation . The ultimate test of the 
possibility for an evaluation of inservice education and 
training to further the career of an evaluator is that it demand 
ne\% levels of performance. To an evaluator, many areas of per- 
formance go beyond the methodology chosen and used. They may 
include new ways of reporting, gaining better understanding 
of the decision making process, designing new ways to integrate 
the evaluation with the instruction or creating new perspec- 
tives on professional development^ New successes or failures 
in the eventual conduct of the study will determine the place' 
of the study in the career of the evaluator. The significance 
of these successes and failures, however, will depend upon the 
extent to which the evaluation was a challenge and required 
some risk-taking in the evaluation design. 

Professional dialogue between policy makers, designers and evaluators 
on the evaluation of inservice education and training programs will depend 
largely on the preparation of the evaluator. In preparing for dialogue, 
the evaluator must be able to articulate the range of evaluation possi- 
bilities and their respective feasibility given the context of the inser- 
vice education arid training program. The evaluator must be prepared to 
talk about evaluation on both a theoretical as well as a practical level 
with the policy makers and designers. They, too, have* theories of evalu- 
ation, of science, of knowledge and its relation to policy. They, too, 

. have p rac t ical experience with.. past .evaluations. In .addition, to addres- 

sing them on crucial issues relating to the evaluation design, the 
evaluator must also be prepared to address the crucial issues relating to 
policy making and to designing of the instruction in inservice education 
and training, just as theories and underlying assumptions about evalua- 
tion are held by the policy makers and designers, so, too, evaluators 
have theories and underlying assumptions about policy making and design- 
ing. These theories and assumptions are relevant co the eventual conduct 
of the evaluation and should be open for critical analysis. The evalu- 
ator must not only be prepared to discuss evaluation with respect towards 
others' views" of evaluation, the evaluator must also be prepared to earn 
the respect of others in discussingtheir professional fields of 
policy making and instructional design. 



In conclusion, recommendations have been given to solicy 
makers, designers and evaluators of - inservice education and 
training functioning in their ov/n work contexts* The recom- 
mendations are given as suggestions for each role preparing 
to engage in dialogue with the others on the evaluation of 
inservice education and training. Recommendations have not 
been .given. onJiow. to perform bettor -evaluation- studies* - 
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It has instead been argued that better evaluations of 
inservice education and training will depend upon better 
dialogue between policy makers, designers and evaluators 
in the context of the specific inservice education and 
training program being evaluated. Furthermore, for this 
dialogue to occur, there must be significantly more 
attention given by the major parties to their own inter- 
pretations of the intentions of the inservice education 
and training programs, to their own underlying assumptions 
about professional development, school change and social 
reform as well as how evaluation can serve these intentions, 
and to their previous experiences- and understanding. This 
includes their understanding not only of evaluation and of 
inservice education and training but a host of related 
issues as well such as their views of science and ways 
of knowing. Figure 7, below, summarises the recommendations 
for the preparation for dialogue between policy makers, 
program designers and evaluators. 
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RECOMMENDATIONS TO 
POLICY MAKERS 



RECOMMENDATIONS TO 
PROGRAM DESIGNERS 



RECOMMENDATIONS TO 
EVALUATORS 



for placing the evaluation into context: 



Articulate the reason you support 
inservice education and training 
(to stimulate professional devel- 
opment, to improve school prac- 
tice, to implement social policy),- 

Consider the efficacy of the 
evaluation to the inservice ed- 
ucation and training context. 

Consider the issues emanating 
from the realities of an evalua- 
tion of inservice education and 
trainings 

Determine where your responsi- 
bilities and continued involve- 
ment may be the most useful in 
the analytic process of the eval- 
uation and in the resolution of 
issues emanating from the reali- 
ties of the evaluation. 



Identify the intent of the in- 
service education and training 
(to stimulate professional de- 
velopment, to improve school 
practice, to implement social 
policy). 

Consider the efficacy of eval- 
uation not only to the instruc- 
tional design but to other is- 
sues suggested by the INSET 
context. 

Consider each of the three lev- 
els of evaluation reality: Its 
bureaucratic context, its method' 
ology and participatory involve- 
ment. 

Determine what your involvement 
in the evaluation may be. 



for ccuironting our understanding: 

Ask yourself what would happen if 
you suspended your belief in sci- 
ence and increased your belief in 
a process whereby information is 
negotiated. 



State ycur assumptions about 
the instructional aspects of 
inservics education and traininc. 



Hake clear your interpretation 
of the reasons for inservice 
education and training program, 
specify which you consider pri- 
orities for investigation. 

Place the inservice education \ 
and training into its probable 
setting (single school, multi- 
ple school, ad-hoc). 

Analyze your approaches to the 
three mundane realities to e- 
valuating inservice education 
and training (its bureaucratic 
context, choice of methodology 
and participation). 

Consider how the evaluation can 
be policy relevant, illuminative 
and participatory. 

Place- your vi ew o f the evaluation 
of this inservice education and 
training program into your pro- 
fessional career. 



State what you have done before 
in educational vocation. 

Place what you have done before 
into a wider context. 



I 



Construct your own view of policy 
and the relationship of informa- 
tion to policy. 

List the Alternatives you see 
to inservice education and 
training. 

Ask yourself if. you can doubt 
your own rhetoric for choosing 
inservice education and training. 



If professional development is 
an intent of the inservice edu- 
cation and training, state your 
visions of what professional 
'development may be. 

State the educational theories 
that influence your inservice 
education and training design. 

State your own theories of how 
educational evaluation relates 
to instructional design. 

Be articulate and clear about 
what you may need to know to 
design inservice education 
and training better. 



for confronting others' understanding: 



Be very selective in who is 
chosen to evaluate inservice 
education and training. 

Look at what the evaluators 
have done in the past. 

Do not accept all the solemn 
pronouncements of evaluators 
towards their methodologies. 



Be prepared to address other 
than your instructional 
intentions. 

Relate your priorities for 
information to those of the 
INSET policy maker. 



State your own assumptions 
about inservice education 
and training. 

Review your own view of science 
and its relationship to the 
kinds of investigations you can 
probably perform in evaluating 
an INSET program. 

Identify your own bureaucratic 
context and what it may mean to 
what you can do and how you carl 
approach an INSET evaluation. 

Try to imagine what the most 
simple and creative ways to 
evaluate inservice education and 
training might be. 



Know the policies in which the 
inservice education and train- 
ing is embedded. 

Prepare to understand the policy 
makers and instructional design- 
ers. 
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Assess the potential evaluators. Determine to the extent to which 

the significance of the policies 
and the challenge of the instruct 
tional design may stretch your 
own cs abilities in evaluation, 



Figure 7. Summary of Recommendations to Policy Makers, Program Designers and Evaluators 
in Preparing for Dialogue on the Evaluation of Inservice Education and Training 
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